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ABSTRACT 


The  National  Preparedness  Goal  directs  that  preparedness  efforts  for  major  events 
should  be  developed  through  a  capability-based  planning  process.  The  ability  to  perforin 
a  valid  and  reliable  capability  assessment  is  necessary  to  guide  proper  allocation  of 
resources  in  the  capability-based  planning  process.  Capability  assessment  is  also 
necessary  to  validate  and  measure  the  impact  of  investments.  This  thesis  examines  a  self- 
assessment  methodology  that  may  be  used  to  perform  capability  assessment.  A 
comparative  case  study  methodology  is  employed  to  examine  process  options.  These 
options  might  be  utilized  to  detennine  which  process  provides  the  most  opportunity  for 
data  collection  while  fostering  a  collaborative  environment  that  may  provide 
opportunities  for  sharing  knowledge  among  assessment  group  members.  A  review  of  the 
case  studies  suggests  that  the  subject-matter  experts  employed  in  the  analysis  may  not 
have  the  requisite  knowledge  to  perform  the  assessment.  This  suggested  finding  implies 
problems  with  capability-based  planning  efforts.  The  thesis  makes  recommendations  for 
future  strategies  to  perform  homeland  security  capability  assessments. 
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EXECUTIVE  SUMMARY 


To  meet  the  requirements  of  HSPD  8,  DHS  has  proposed  a  self-assessment 
process  to  perform  the  national  capability  assessment.  This  document  examines  the 
results  from  a  test  deployment  of  the  proposed  system.  The  expected  results  of  the 
deployment  were  the  production  of  data  that  could  be  used  to  measure  the  effect  of 
investments  in  homeland  security  to  provide  a  snapshot  of  preparedness.  DHS  also 
expected  this  methodology  to  enhance  the  knowledge  of  participants  in  the  assessment 
process. 

The  results  of  the  comparative  case  study  suggest  a  flaw  with  a  basic  assumption 
of  the  assessment  process.  The  study  suggests  that  the  locally  appointed  subject-matter 
experts  may  not  be  able  to  employ  the  proposed  methodology  to  produce  valid  and 
reliable  data.  The  study  did  show  an  undercurrent  of  preparedness  activities  that  were  not 
captured  by  the  assessment  tool,  but  suggest  collaborative  homeland  security  planning 
efforts  exist  at  the  local  level. 

The  findings  are  significant  because  they  undermine  the  capability-based  planning 
process.  Capability-based  planning  develops  investments  based  on  gaps  and  needs 
identified  in  an  assessment.  Faulty  assessments  will  reduce  national  preparedness  by 
causing  our  decreasing  resources  to  be  invested  to  close  gaps  that  may  or  may  not  exist. 

This  thesis  recommends  that  DHS  take  steps  that  create  an  understanding  of  how 
capabilities  are  implemented  at  the  local  level,  before  efforts  are  made  to  assess  those 
capabilities.  The  document  also  recommends  that  the  National  Response  Plan,  HSPD  5 
and  HSPD  8  be  coordinated  so  that  requirements  and  guidance  regarding  capabilities  be 
made  consistent. 
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I.  INTRODUCTION 


A.  PROBLEM  STATEMENT 

The  National  Preparedness  Goal  ( NPG )  as  set  forth  in  Homeland  Security 
Presidential  Directive  #  8  (HSPD  8)  is  a  plan  to  ensure  that  our  country  is  prepared  for  a 
catastrophe  and  positioned  to  prevent  a  tragedy.  Through  the  designation  of  capabilities- 
based  planning  and  universal  tasks,  the  goal  sets  a  path  to  preparedness.  To  support  the 
goal,  federally  funded  grant  programs  provide  resources  and  incentive  to  local 
implementation  of  the  plan. 

A  solid  foundation  is  required  for  any  serious  undertaking.  In  capabilities-based 
planning,  the  foundation  is  performance  of  both  risk  and  capabilities  assessments.  This 
thesis  focuses  on  one  of  these  pillars:  the  capability  assessment.  If  the  capability 
assessment  is  not  adequate,  everything  that  follows  is  built  on  a  broken  foundation.  In 
educational  policy  development  the  focus  on  foundations  is  understood.  For  example,  the 
foundational  skills  of  reading  and  writing  are  taught  prior  to  critical  analysis  of  poetry. 

Why  then,  have  we  ignored  the  foundation  of  our  capabilities-based  planning 
cycle?  The  last  national  capabilities  assessment  was  conducted  in  2003,  yet  we  have 
gone  on  to  create  and  fund  implementation  of  countless  homeland  security  programs. 
These  are  commendable  accomplishments,  but  —  lacking  an  interwoven  capability 
assessment  —  the  accomplishments  are  built  on  a  poor  foundation.  Without  perfonning 
assessments  as  part  of  the  investment  process,  it  is  unknown  how  far  those  investments 
have  moved  the  nation  toward  meeting  preparedness  goals. 

My  experience  with  homeland  security  efforts  in  the  National  Capitol  Region 
(NCR)  demonstrates  assessment  to  be  a  factor  in  the  success  or  failure  of  efforts  to 
implement  the  NPG.  The  inability  to  measure  the  effects  of  investments  that  are 
implemented  as  part  of  the  NPG,  hinders  the  ability  to  gain  funding  for  programs  that 
could  bring  safety  and  security. 
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The  nation’s  previous  attempts  at  assessment  did  not  produce  changes  in  behavior 
or  create  plans  that  caused  improvements  in  homeland  security.  The  2003  National 
Capabilities  Assessment  was  a  paper  lion,  consuming  personnel  resources  and  providing 
little  useful  feedback.1 

The  2005  and  2006  Homeland  Security  Grant  Application  processes  required  the 
performance  of  a  gaps  analysis  based  on  the  NPG.  In  the  NCR,  the  2005  gaps  analysis 
was  completed  with  technical  assistance  from  the  Department  of  Homeland  Security. 
This  effort  resulted  in  three  consecutive  days  of  proctored  discussion  about  existing  gaps. 
The  focus  group  participants  turned  the  discussion  to  defending  existing  grant  funding 
requests.  The  result  of  this  effort  was  a  report  in  which  identified  gaps  that  aligned  with 
funding  requests.  This  effort  did  not  provide  data  on  existing  gaps,  nor  did  the 
assessment  process  produce  an  environment  that  encouraged  open  and  frank  discussion.2 

This  faulty  foundation  for  the  regional  grant  applications  was  a  contributing  factor 
in  the  low  marks  the  submissions  received  for  relating  funding  priorities  to  existing  gaps 
in  the  NPG.  The  ultimate  result  was  reduced  funding  and  reduced  ability  to  improve 
preparedness. 

In  2006,  these  inefficient  top-down  processes  resulted  in  the  local  development  of 
a  self  assessment  for  capabilities  that  fell  under  the  responsibility  of  Emergency  Support 
Function  #4,  Firefighting.  This  effort  was  well-received  within  the  discipline,  and 
produced  data  and  discussion  that  helped  to  create  stronger  homeland  security  capability 
assessment.  By  strengthening  the  foundation,  the  product  improved. 

Members  of  Emergency  Support  Function  (ESF)  13,  Public  Safety  and  Security 
Annex,  were  exposed  to  the  self-assessment  in  their  supporting  roles  to  ESF  4.  This 
exposure  caused  them  to  adopt  the  self  assessment  and  deploy  it  for  target  capabilities  in 
which  they  were  the  lead  agency.  Both  ESF  4  and  ESF  13  felt  this  self-assessment 


1  GAO  Report,  05-121,  “Homeland  Security,  Management  of  First  Responder  Grant  Programs  Has 
Improved,  but  Challenges  Remain,”  February  2005,  17 

2  This  paragraph  is  a  summary  of  observations  made  by  the  author  after  participation  in  the  2005 
Homeland  Security  Grant  Process.  The  author  was  participating  in  the  assessment  process  as  the  leader  of 
the  Metropolitan  Washington  Council  of  Governments,  Fire  Chiefs  Hazardous  Materials  Subcommittee. 
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process  triggered  an  enhanced  understanding  of  capability  status.  This  understanding 
resulted  from  breaking  the  assessment  processes  into  manageable  modules  and  rapidly 
providing  assessment  results.  Planners  and  decision  makers  used  the  results  from  the 
assessment  to  obtain  an  understanding  of  the  whole  capability  —  not  just  specific  pieces. 

This  local  assessment  solution  was  unfinished,  but  it  created  an  environment  that 
sought  to  follow  the  preparedness  cycle  and  focus  on  a  foundational  assessment  process 
to  inform  investments.  Members  of  multiple  disciplines  exposed  to  the  survey  credited 
the  survey  process  with  inspiring  them  to  become  more  familiar  with  the  elements  of 
their  capability.  The  results  of  the  survey  gave  them  an  understanding  of  the  status  of 
their  plans  and  programs  in  relation  to  meeting  the  requirements  of  the  NPG.  This 
capability  survey  effort  was  a  step  in  the  right  direction.  By  providing  a  definition  of 
current  capability,  our  small  homeland  security  community  sought  cross-discipline 
solutions  to  gaps  —  instead  of  chasing  funding  for  single-discipline  programs  that 
already  had  high  levels  of  implementation. 

At  the  national  level,  the  Target  Capabilities  Pilot  Assessment  Working  Group 
has  been  at  work  developing  the  next  National  Capabilities  Assessment  with  similar 
characteristics  to  the  locally  developed  solution  described  above.  This  working  group 
developed  the  Target  Capability  Self  Assessment  Tool  and  process  described  in  this 
thesis. 

B.  RESEARCH  QUESTION 

A  DHS  working  group  has  proposed  a  methodology  using  locally  selected 
subject-matter  experts  to  perfonn  capability  assessments.  Can  a  portion  of  this 
assessment  methodology,  designed  to  create  collaboration,  be  altered  without  affecting 
the  assessment  of  capability  status?  What  is  the  effect  of  the  process  changes  on  the 
desired  additional  benefits:  enabling  and  improving  collaboration  among  homeland 
security  professionals? 

In  the  proposed  assessment  methodology,  collaborative  opportunity  is  an  expected 
byproduct  of  the  assessment  process.  In  this  process,  an  assessment  team  reaches  a 
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consensus  opinion  regarding  a  capability  status.  DHS  expects  that  the  consensus  process 
will  facilitate  sharing  of  information  and  the  creation  of  relationships  among  assessment 
team  members.  The  benefits  of  this  process  are  expected  to  include  more  accurate 
assessments  and  the  creation  of  relationships  that  support  future  capability-building 
efforts.  What  is  the  effect  of  different  processes  on  the  expected  collaborative  benefits? 
Does  changing  the  assessment  process,  from  a  consensus  method  to  an  aggregated 
method,  affect  the  expected  benefits  of  information-sharing  and  relationship-building? 

C.  HYPOTHESIS 

The  hypothesis  for  this  research  question  is  that  altering  the  assessment  process 
will  not  play  a  key  role  in  the  results,  and  that  capability  status  assessments  from  the 
different  processes  will  agree.  A  further  hypothesis  is  that  of  the  different  processes,  one 
process  will  be  identified  as  having  provided  more  additional  benefits  than  the  others.  If 
the  hypothesis  is  true,  it  will  result  in  the  ability  to  identify  which  process  provides  the 
greatest  opportunity  to  determine  capability  status  and  provide  additional  benefits  to  the 
assessment  team. 

D.  SIGNIFICANCE  OF  THESIS 

Homeland  Security  is  a  nascent  discipline,  with  dedicated  guidance  and  funding 
initially  occurring  as  a  result  of  the  Oklahoma  City  and  Tokyo  terrorist  attacks.3  The 
terrorist  attacks  of  9/11  rapidly  increased  pressure  on  the  discipline  to  increase  national 
levels  of  preparedness.  The  maturing  process  of  the  discipline  requires  academic  study  of 
methods  used  to  evaluate  the  components  of  the  discipline.  To  date,  there  has  been  some 
research  on  measuring  homeland  security  preparedness  status,  but  this  literature  is  not 
equal  to  the  task  of  understanding  the  intricacies  and  nuances  that  occur  in  the 
performance  of  homeland  security  capability  assessments.4  This  thesis  will  expand  on 
that  body  of  knowledge  by  focusing  on  the  processes  needed  to  perform  a  capabilities 

3  Andy  Mitchell,  “On  Domestic  Preparedness  Efforts,”  Testimony  before  the  Committee  on  the 
Judiciary,  United  States  Senate,  April  20,  1999,  http://iudiciarv.senate.gov/oldsite/42099am.htm  accessed 
August  16,  2007. 

4  Glen  Woodbury,  “Measuring  Prevention,”  Homeland  Security  Affairs  I,  no.  1  (Summer  2005),  10. 
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assessment  in  the  homeland  security  environment.  Specifically,  the  findings  in  this  thesis 
will  examine  how  members  of  the  homeland  security  discipline  are  able  to  perfonn  an 
examination  of  their  field. 

Homeland  Security  practitioners  involved  in  planning  activities  will  benefit  from 
the  development  of  a  framework  for  understanding  an  assessment  process  proposed  for 
use  in  homeland  security.  Ultimately,  this  understanding  of  the  assessment  process  will 
help  decision  makers  when  allocating  resources  to  improve  capabilities. 

The  findings  and  recommendations  included  in  this  thesis  will  help  national 
leaders  understand  capability  assessment  at  the  local  level.  Homeland  security  leadership 
can  use  the  suggested  findings  presented  in  this  thesis  to  develop  capability  assessment 
methodologies  that  meet  both  national  and  local  priorities. 

E.  LITERATURE  REVIEW 

The  terrorist  attacks  of  9/11  and  the  devastating  effects  of  Hurricane  Katrina 
caused  the  nation  to  focus  energy  and  resources  on  how  the  nation  plans  for  and  responds 
to  emergencies.  The  studies,  results  and  ideas  are  incomplete,  but  the  environment  this 
examination  has  wrought  is  producing  new  ways  of  looking  at  how  the  basic  functions  of 
planning  are  performed  in  the  emergency  preparedness  community.  This  thesis  will 
focus  on  the  methodology  of  measuring  capabilities.  This  is  also  an  area  that  has  caused 
angst  for  emergency  preparedness  planners,  as  they  attempt  to  quantify  or  classify 
capability. 

The  starting  point  for  the  literature  review  is  the  government  documents  that  state 
the  requirement  or  assess  the  results  of  the  existing  reviews.  A  capability  review  is 
required  by  HSPD  8,  Section  24  Assessments  and  Evaluations.5  Additional  requirements 
for  the  capability  review  are  found  in  State  Homeland  Security  Program  and  Capability 
Review  Guidebook,  Volume  1.  This  document  defines  minimum  requirements  of  the  state 


5  George  Bush,  Homeland  Security  Presidential  Directive/HSPD-8  (Washington,  DC:  The  White 
House,  2004),  5. 
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capability  review  and  how  the  results  are  used  in  the  grant  process.6  The  accuracy  and 
results  of  the  capability  review  are  indicated  by  the  importance  placed  on  them  in  the 
grant  allocation  process.  The  Department  of  Homeland  Security  specifically  suggests 
that  grant  applications  be  used  to  fund  gaps  in  state  capabilities  as  identified  in  the 
capability  review.7  The  Department  of  Homeland  Security,  Grants  and  Training  Office 
directs  that  scoring  of  grant  applications  will  be  based  on  the  ability  to  clearly  state  a  plan 
to  close  weaknesses  in  state  capabilities.8  Unfortunately,  the  guidelines  to  perfonn  a 
capability  review  are  not  clearly  specified  in  the  capability  review  guidance. 

The  NPG  is  the  roadmap  proposed  by  the  federal  government  to  create  a  nation 
prepared  to  prevent,  respond  and  recover  from  natural  disasters  and  terrorist  attacks.  The 
NPG  has  three  components:  National  Planning  Scenarios,  the  Target  Capabilities  List  and 
the  Universal  Task  List.  Within  the  NPG,  capability-based  planning  is  described  as  the 
preferred  method  to  developing  investments  to  reduce  the  risk  from  all  types  of  hazards.9 

The  NPG  provides  a  basis  to  assess  preparedness  to  set  priorities  for  the  most 
effective  use  of  limited  resources.10  This  basis  is  the  Target  Capability  List  (TCL).  The 
TCL  is  a  list  of  thirty-six  capabilities  that  state  and  local  governments  are  expected  to 
create  and  maintain.  Not  every  jurisdiction  will  be  required  to  maintain  each  capability; 
the  decision  on  which  capabilities  to  implement  should  be  decided  based  on  the  risk  and 
circumstances  present  within  an  area  of  hazard.11  To  implement  these  broadly  defined 
capabilities,  coordination  is  needed  between  government,  citizens  and  the  private  sector. 


6  Department  of  Homeland  Security,  State  Homeland  Security’  Program  and  Capability  Review 
Guidebook,  Volume  1  (Washington,  DC:  Department  of  Homeland  Security,  2005),  4. 

7  Matt  Mayer,  letter  to  Edward  Reiskin,  October  21,  2005. 

8  Tracie  Henke,  “Application  Review  and  Scoring  Approach  for  Fiscal  Year  (FY)  2006  Homeland 
Security  Grant  Program.”  G&T  Information  Bulletin,  no.  202  (February  2006):  A-l. 

9  Department  of  Homeland  Security,  National  Preparedness  Goal  (Washington,  DC:  Department  of 
Homeland  Security,  2005),  4. 

10  Department  of  Homeland  Security,  Target  Capabilities  List,  Version  2.0  (Draft)  (Washington,  DC: 
Department  of  Homeland  Security,  2005),  5. 

11  Department  of  Homeland  Security,  National  Preparedness  Goal  (Washington,  DC:  Department  of 
Homeland  Security,  2005),  7. 
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Amy  Donahue  points  out  controversy  in  the  effectiveness  of  the  TCL  as  a 
measuring  tool.  She  argues  that  the  TCL  is  too  large  and  that  it  assumes  common 
behaviors  across  emergency  response  disciplines.  As  a  result  of  this  generality, 
capability  requirements  may  vary  across  different  target  capabilities  and  obscure  the  need 
to  focus  on  local  coordination  and  shared  capabilities. 12 

Current  work  by  DHS  is  focused  on  formulating  strategy  to  conduct  capability 
assessments  that  are  systemic,  repeatable,  useful  and  consistent  in  what  they  measure  and 
the  benefits  the  assessment  produce.  The  assessment  goals  are  twofold:  to  gather  data  on 
the  nation’s  level  of  preparedness  and  to  facilitate  local  understanding  of  preparedness.  In 
Florida,  a  recent  pilot  assessment  identifies  issues  that  should  be  addressed  in  order  for  a 
capability  assessment  to  help  regions  develop  programs  that  are  consistent  with  strategic 
plans  and  provide  a  database  to  measure  the  value  of  investments  in  those  programs  and 
plans  over  time.13  The  report  is  implying  that  an  assessment  tool  alone  cannot  create  the 
opportunity  to  infer  collaboration  between  disciplines,  and  the  process  for  utilizing  that 
tool  is  as  important  as  the  tool  itself  in  the  final  use  of  the  assessment  data. 

The  ability  to  measure  collaboration  is  of  equal  importance  because  accurate 
measures  of  these  complex  capabilities  require  multiple  sources  of  information  from  all 
participants.  The  failure  to  include  one  agency  in  the  assessment  process  could  lead  to  an 
inaccurate  assessment.  In  this  manner,  each  capability  requirement  must  work  together 
with  the  others  to  equal  a  capability  that  is  more  than  the  sum  of  the  separate  parts.  The 
extent  to  which  these  separate  parts  work  together  to  achieve  the  greater  output  is 
collaboration.  Understanding  the  collaboration  effort  is  therefore  important  to 
understanding  the  impact  of  the  measured  capabilities  on  preparedness. 

Any  method  used  in  the  emergency  preparedness  community  could  usefully  be 
compared  to  established  capability  or  program  assessment  systems  used  elsewhere  in 
government  to  assure  wide  acceptance.  The  Program  Evaluation  Profile  tool,  developed 

12  Amy  Donahue,  Concerns  about  the  Department  of  Homeland  Security ’s  Target  Capabilities  List 
Instrument  and  Process  (talking  points  paper  presented  to  Interagency  Board  Chair,  DRAFT,  June  2006). 

13  Department  of  Homeland  Security,  Discussion  on  Changes  to  the  Pilot  Capabilities  Self  Assessment 
Tool  Pilot  Capabilities  Assessment:  Methodology  Meeting  (Washington,  DC:  Department  of  Homeland 
Security,  October  25,  2006),  slides  10,  11. 
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by  the  Department  of  Labor,  is  a  system  for  measuring  safety  programs  in  an  effort  to 
make  the  workplace  safe.  Because  this  tool  seeks  similar  results  to  the  emergency 
preparedness  field,  any  similarities  will  provide  an  area  of  agreement  within  government. 
There  are  other  standardized  evaluation  tools  and  methodologies  in  the  Government 
Accounting  Office  and  Office  of  Management  and  Budget  (OMB).  OMB  uses  the 
Program  Assessment  Rating  Tool  (PART)  which  also  uses  an  assessment  tool. 

Theodore  Poister  and  John  Thomas,  of  Georgia  State  University,  examine  the 
possibility  of  using  the  methodology  from  James  Surowieki’s  “Wisdom  of  Crowds”  to 
assess  citizen  perception  of  the  highway  department.  In  their  presentation  “The 
‘Wisdom  of  Crowds’:  Learning  from  Administrators’  Predictions  of  Citizen  Perceptions,” 
the  authors  examine  two  case  studies.  They  conclude  that  the  theory  has  merit  in 
discovering  unknown  information  and  is  suited  to  specific  types  of  crowds.  The  concept 
of  discovering  unknown  information  has  parallels  to  discovering  whether  collaboration  is 
occurring  within  a  target  capability.  The  lessons  from  this  process  will  be  addressed  in 
the  thesis.  The  small  size  of  this  comparative  case  study  is  problematic,  but  it  is  of 
interest  because  the  findings  are  similar  to  the  predictions  made  by  Surowieki.14 

Recent  literature  examines  measurements  of  homeland  security  capabilities.  Glen 
Woodbury’s  article,  “Measuring  Preparedness”,  advocates  an  approach  at  the  state 
government  level  to  measure  preparedness  outputs.  Woodbury’s  scale  applies  broad 
questions  about  prevention  outcomes  to  determine  critical  gaps  in  knowledge  about  the 
effort  being  applied  to  close  those  gaps.  He  argues  that  this  method  can  be  used  in  the 
prevention  field  and  to  predict  consequence.  The  assessment  tool  he  uses  is  similar  to  the 
one  proposed  in  this  thesis,  but  the  methodology  of  his  assessment  is  different.15 

The  literature  relating  to  collaboration  in  homeland  security  is  more  diverse.  In 
their  report  on  building  collaborative  capacity,  Thomas,  Hocevar  and  Jansen  describe  the 
benefits  of  collaboration  as  cost  savings,  improved  decision  making  and  maximizing  the 

14  Theodore  Poister  and  John  Thomas,  “The  ‘Wisdom  of  Crowds’:  Learning  from  Administrators’ 
Predictions  of  Citizen  Perceptions”  (presentation  at  the  Public  Management  Research  Conference,  Los 
Angeles,  CA,  September  29,  2005),  21. 

15  Glen  Woodbury,  “Measuring  Prevention,”  Homeland  Security  Affairs  I,  no.  1  (Summer  2005):  10, 
http://www.hsaj.org/7articleM.L7  (accessed  September  13,  2006). 
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efficiency  of  scarce  resources.  The  authors  argue  that  improving  collaboration  is 
especially  important  in  instances  when  organizations  must  work  together  in  support  of  a 
common  goal.16  The  benefits  of  collaboration  that  Thomas,  Hocevar  and  Jansen  have 
identified  will  need  to  be  examined  as  effective  measures  of  collaboration  in  the 
homeland  security  community.  It  may  be  possible  that  collaboration  exists,  but  it  does 
not  produce  the  expected  results,  or  that  no  collaboration  exists  and  the  identified  outputs 
are  met. 

A  Government  Accountability  Office  (GAO)  report  examines  collaborative 
efforts  in  the  federal  government.  The  report  identifies  areas  that  can  be  examined  to 
assess  collaboration.  These  areas  are:  definition  of  common  outcome,  joint  strategies, 
leveraging  resources,  defined  roles  and  policies,  and  accountability.17  The  GAO  issued  a 
follow-up  report  that  found  the  government’s  efforts  at  homeland  security  collaboration 
to  be  at  a  high  risk  for  failure.18  The  definition  and  impact  of  this  failure  will  need  to  be 
examined.  There  may  be  instances  where  a  lack  of  collaboration  is  acceptable  to  the 
parties  involved,  and  the  capability  levels  achieved  by  the  parties  are  acceptable  to  their 
communities. 

The  primary  gap  in  the  literature  is  that  DHS  has  yet  to  publish  a  capability 
assessment  strategy.  Initial  efforts  by  DHS  have  produced  pilot  capability  assessments, 
but  how  useful  these  pilots  have  been  has  not  been  fully  examined.  The  purpose  of  this 
thesis  is  to  explore  how  capability-assessment  efforts  may  be  improved  by  focusing  on 
the  process.  Findings  from  the  thesis  will  be  used  to  make  recommendations  to  inform 
capability  assessment  strategy  development. 


16  Gail  Thomas,  Susan  Hocevar  and  Erik  Jansen,  A  Diagnostic  Approach  to  Building  Collaborative 
Capacity  in  an  Interagency  Context  (Monterey,  CA:  Naval  Postgraduate  School,  2006),  2. 

17  United  States  Government  Accountability  Office,  Results-Oriented  Government:  Practices  that 
Enhance  and  Sustain  Collaboration  Among  Federal  Agencies,  GAO-05-15  (Washington,  DC:  GAO,  2005), 
1. 

18  United  States  Government  Accountability  Office,  High  Risk  Series  and  Update,  GAO-05-207 
(Washington,  DC:  GAO,  2005),  1. 
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II.  ARGUMENTS 


The  core  claim  of  this  thesis  is  that  the  process  for  performing  capability 
assessments  in  homeland  security  must  be  studied  to  determine  which  methodology 
provides  the  most  reasonable  investment  of  resources,  based  on  expected  outcomes.  The 
expected  outcomes  of  the  assessment  are  that  it  creates  a  snapshot  of  current  capability 
status  and  causes  additional  benefits  to  be  derived  from  the  process  of  perfonning  an 
assessment. 

The  value  of  capability  assessments  in  homeland  security  results  from  their 
foundational  role  in  capability-based  planning.  Understanding  the  limits  of  capability 
assessment  benefits  local  officials  who  make  investment  decisions  based  upon  the  results 
of  an  assessment. 

HSPD  8,  the  NPG,  states  that  the  nation  can  become  prepared  by  developing 
appropriate  threat-based  capabilities.19  The  Goal  further  defines  capabilities-based 
planning  as  the  chosen  mechanism  for  guiding  Homeland  Security  evolution.  The  first 
step  in  capability-based  planning  is  assessing  what  capabilities  exist,  which  capabilities 
do  not,  and  how  to  measure  changes  over  time.20  Focusing  on  the  first  step  in  the  process 
will  improve  the  effect  of  planning  efforts  at  all  stages  of  the  process  and  is,  therefore,  the 
focus  of  this  thesis.  21 

To  satisfy  the  capability-based  planning  demands,  an  assessment  process  should 
be  trusted  by  those  charged  with  using  the  results.  Trust  in  the  assessment  will  allow  a 
confidence  that  investments  made  to  correct  gaps  identified  by  the  assessment  are 
defensible  and  proper  use  of  taxpayer  funds.  This  thesis  will  examine  assessment 
strategies  and,  using  data  gained  from  a  series  of  pilot  assessments,  recommend  a  strategy 
for  conducting  assessments  that  satisfies  the  needs  of  the  homeland  security  community. 

19  George  Bush,  Homeland  Security  Presidential  Directive/HSPD-8  (Washington,  DC:  The  White 
House,  December  2004),  2. 

20  Paul  Davis,  Analytic  Architecture  for  Capability-based  Planning,  Mission  System  Analysis,  and 
Transformation  (Santa  Monica,  CA:  RAND,  2002),  xii. 

21  Ibid.,  xii. 
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In  its  short  history,  DHS  has  failed  to  develop  a  relationship  of  trust  with  local 
officials.  When  talking  about  the  collaborative  and  information-sharing  processes  at 
DHS,  Representative  Bennie  Thompson  states,  “The  department  must  start  querying  state 
and  local  organizations  on  their  existing  initiatives  and  develop  better  collaborative  and 
coordinating  tools.”22  The  faults  in  this  relationship  are  potential  barriers  to  obtaining  the 
necessary  data  from  the  assessment  process.  To  ensure  the  assessment  creates  data  that  is 
systematic,  repeatable,  consistent  and  useful,  it  should  be  collected  in  a  manner  that 
recognizes  a  lack  of  trust  by  local  government  of  DHS.  To  improve  trust,  an  assessment 
process  that  creates  collaboration  by  identifying  local  data  needs  are  transparent,  and 
shows  the  benefits  of  engaged  participation,  will  allow  the  development  of  strategic 
planning  processes  to  improve  homeland  security.  One  way  this  process  will  improve 
planning  is  to  provide  a  database  to  measure  improvement  in  preparedness  over  time. 
This  process  should  also  create  an  understanding  of  the  interdependencies  at  local  and 
regional  levels  required  to  achieve  sufficient  levels  of  preparedness. 

This  thesis  examines  how  collaboration  in  the  assessment  process  produces 
outcomes,  or  impacts  that  improve  homeland  security  preparedness.  The  discovery  of 
capability  gaps  that  otherwise  would  not  have  been  discovered  as  a  result  of  collaboration 
in  the  assessment  process  is  an  example  of  the  benefit  of  collaboration.  If  collaboration 
does  produce  beneficial  outcomes,  then  the  starting  point  for  understanding  these  benefits 
is  the  assessment  process. 

One  of  the  DHS  goals  in  the  assessment  process  is  to  create  an  understanding  of 
preparedness  activities  at  the  local  level.  If  collaboration  produces  benefits  to 
preparedness,  then  collaboration  should  be  included  at  initial  stages  of  the  planning 
efforts:  the  assessment  process.  Inclusion  of  collaborative  effort  at  the  earliest  stages  may 
create  opportunities,  through  the  assessment  process,  for  future  collaboration  in  support 
of  an  established  national  plan  for  preparedness. 


22  Bennie  Thompson,  “Department  of  Homeland  Security  churns  out  poorly  developed  schemes,”  The 
Hill.  June  5,  2007,  http://thehill.com/leading-the-news/department-of-homeland-security-churns-out- 
poorly-developed-schemes-2007-06-05.html  (accessed  March  16,  2007). 
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An  area  of  challenge  is  that  DHS  has  created  a  problem  by  adopting  capability- 
based  planning  as  the  strategy  for  securing  the  homeland.  This  argument  is  supported  by 
the  adoption  of  effects-based  planning  by  the  military  and  supported  by  the  Office  of 
Management  and  Budget.  This  challenge  has  a  corollary  argument:  The  planning  efforts 
are  too  complex  and  the  TCL  is  given  too  much  weight  in  defining  the  outcomes  of 
planning  efforts.  These  challenges  would  argue  that  relying  on  state  and  local 
government  to  produce  data  —  data  that  meets  the  stated  goal  of  supporting  strategic 
planning  efforts  and  assessing  preparedness  —  is  impossible  because  of  the  variety  of 
communities  in  our  nation  and  the  complexity  of  the  TCL. 

The  following  comparative  case  study  in  a  large  urban  area  focuses  on  these 
controversies.  For  instance,  the  research  suggests  the  extent  to  which  data  gathered  in  an 
assessment  process  is  systematic,  repeatable,  consistent,  and  useful  to  a  complex  strategic 
planning  process.  These  analyses  also  lead  to  a  discussion  of  strategies  to  improve 
homeland  security  preparedness  by  improving  the  assessment  process.  Capability-based 
planning  efforts,  for  example,  may  have  more  value  if  they  are  incorporated  into  an 
effects-based  planning  process  that  focuses  more  directly  on  how  specific  capabilities  — 
even  if  they  vary  across  jurisdictions  —  produce  desired  outcomes. 
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III.  METHODOLOGY 


A.  OVERVIEW 

The  methodology  employed  in  this  thesis  is  a  specific,  localized  test  of  the  effect 
of  altering  a  portion  of  the  assessment  process.  The  comparative  case  study  is  designed 
to  examine  whether  and  how  an  assessment  process  produces  measures  of  capabilities 
and  adds  value  to  participants.  The  methodology  employed  in  the  study  will  include 
using  a  portion  of  the  Self-Assessment  Tool  developed  by  a  DHS  Pilot  Capability 
Working  Group.  This  tool  is  designed  to  measure  regional  progress  toward  meeting  the 
NPG.23  Additionally,  the  working  group  expects  that  through  the  assessment  process 
additional  benefits  for  participants  will  be  created.  These  benefits  include  opportunity  for 
future  collaboration  in  developing  homeland  security  capabilities  and  an  opportunity  for 
participants  to  increase  their  knowledge  of  local  capabilities. 

The  study  will  utilize  focus  groups  composed  of  existing  multi-agency,  multi- 
jurisdictional  committees  from  the  National  Capital  Region.  These  groups  will  employ 
different  processes  to  conduct  a  self-assessment  of  regional  capability.  The  processes  to 
be  employed  are  designed  to  create  different  opportunities  for  convenience  and 
collaboration  as  assessment  group  members  work  toward  completion  of  the  self- 
assessment  tool. 

Developing  a  methodology  for  understanding  the  effect  of  process,  as  it  relates  to 
opportunity  for  participation  by  subject-matter  experts,  is  important  to  ensuring 
participation  of  subject-matter  experts.  Multiple  demands  on  the  schedules  of  the 
subject-matter  experts  limit  opportunity  for  their  participation  in  the  self-assessment 
process.24  Developing  a  process  that  allows  for  greater  participation  by  homeland 
security  professionals  should  cause  more  accurate  and  complete  assessments  to  be 

22Lauren  Fernandez,  Pilot  Capabilities  Assessment  Orientation  Brief,  Pilot  Capabilities  Assessment 
Working  Group  Meeting  (DHS:  Washington,  DC,  February  22,  2006),  slide  8. 

24  Lauren  Fernandez,  “Pilot  Version  2.0:  Minnesota  Update  and  Discussion,”  Pilot  Capabilities 
Assessment  Working  Group  Meeting  (DH:  Washington  DC,  June  27,  2006),  slide  12. 
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conducted.  Factors  affecting  opportunity  to  participate  include  the  time  needed  to 
complete  the  assessment  process  and  the  convenience  of  scheduling  the  time 
commitment. 

The  data  collected  in  the  study  will  be  analyzed  to  compare  reliability  of  results 
across  different  processes.  Structured  interviews  with  assessment  participants  will 
examine  opinions  for  the  utility  of  the  assessment  data  and  discover  collaborative  benefits 
produced  by  the  assessment  processes. 

B.  ASSESSMENT  PROCESSES 

In  the  study,  three  different  assessment  processes  were  employed  to  complete  a 
self  assessment.  Each  of  the  assessment  processes  utilized  the  same  measures  to  rate 
status  of  the  Mass  Surge  and  Hazardous  Materials  and  Weapons  of  Mass  Destruction 
Response  and  Decontamination  capabilities.  Each  of  these  processes  is  designed  to 
provide  varied  opportunities  for  collaboration. 

Assessment  Process  1  creates  the  most  convenient  opportunity  for  participation 
because  the  participant  controls  both  the  time  and  the  location  of  the  assessment.  This 
assessment  process  creates  the  least  opportunity  for  collaboration  and  discovering 
knowledge  about  the  capability  because  the  participants  can  choose  to  work  alone  to 
complete  the  assessment.  In  this  process,  assessment  group  members  will  be  directed  to 
complete  the  self  assessment  by  a  date  two  weeks  in  advance.  The  ratings  of  all 
participants  will  be  aggregated  into  a  final  rating  as  to  the  status  of  the  measured 
capability.  The  final  output  of  Assessment  Process  1  is  a  report  on  the  capability  status. 
This  process  will  be  employed  in  the  assessment  conducted  by  Assessment  Group  1 . 

Assessment  Process  2  creates  a  mixed  opportunity  for  convenience  and 
collaboration.  Step  one  of  the  process  is  to  perform  Assessment  Process  1 .  Step  two  of 
the  process  is  added  after  aggregation.  Group  members  will  be  presented  the  aggregated 
data  and  provided  the  opportunity  to  build  consensus  to  a  final  rating.  The  choices  for 
the  assessment  group  will  be  to  accept  the  rating  as  accurate,  or  change  the  rating  to 
reflect  the  actual  status.  This  process  increases  the  opportunity  to  collaborate  on  the 
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results  of  the  assessment  by  initiating  discussion  on  capability  status  to  reach  a  consensus 
rating.  Members  of  Assessment  Group  2  will  utilize  this  process  to  perform  their 
assessment. 

Assessment  Process  3  employes  a  process  to  complete  the  self  assessment  that 
limits  opportunity  for  participation  by  requiring  the  assessment  group  member  to  dedicate 
a  specific  block  of  time  at  a  specific  location  to  complete  the  process.  The  benefit  to  this 
process  is  that  the  design  increases  collaborative  opportunity  for  participants  by  causing 
them  to  work  together  to  reach  a  consensus  assessment  rating  for  each  measure. 
Assessment  Group  3  and  4  will  use  this  process. 

C.  ASSESSMENT  GROUPS 

Evaluating  how  process  affects  the  outcomes  requires  that  similar  assessment 
groups  and  a  similar  assessment  tools  are  employed  in  different  assessment  processes. 
This  research  will  use  existing  regional,  multidiscipline  committees  to  conduct  the  self 
assessment  utilizing  different  processes. 

1.  Assessment  Group  1 

The  National  Capitol  Region,  Mass  Decontamination  Working  Group  is 
comprised  of  representatives  of  fire  departments,  emergency  medical  services,  police 
departments,  health  departments,  hospitals,  hazardous  materials  teams,  emergency 
management,  technology  offices,  and  transportation  offices  from  state  and  local 
governments.  This  mission  of  this  group  is  to  develop  and  implement  policies  and  plans 
to  improve  Mass  Decontamination  capabilities  within  the  National  Capital  Region. 

Members  of  this  group  are  decision  makers  from  their  respective  agencies  who 
have  been  tasked  with  developing  innovative  solutions  to  complex  problems.  It  is  the 
expectation  of  regional  leaders  that  members  appointed  to  this  committee  have  intimate 
knowledge  of  Homeland  Security  issues  in  the  National  Capital  Region.  These  members 
should  also  be  able  to  understand  how  issues  of  one  discipline  or  jurisdiction  affect  other 
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disciplines  or  jurisdictions.  Many  of  the  group  members  represent  their  agencies  in  the 
Homeland  Security  Grant  Process  and  are  leaders  of  other  related  committees  and 
organizations. 

Assessment  Group  1  will  be  employed  to  examine  a  process  that  eliminates  the 
need  for  members  to  work  together  when  completing  the  assessment.  Time  commitment 
is  a  problem  for  senior  level  managers  of  any  type  of  organization.  Development  of  a 
method  that  reduces  time  impact  by  increasing  flexibility  may  increase  participation  in 
the  assessment  process  and  improve  the  accuracy  of  results  by  encouraging  the 
participation  of  the  target  audience. 

To  examine  the  reduced  time  processes,  group  members  will  complete  the 
assessment  tool  individually.  When  completed,  the  assessment  tool  will  be  returned  to  a 
coordinator  who  will  compile  the  results  of  the  assessment.  The  result  will  be  the 
average  rating  to  each  measure  by  the  individual  group  members.  In  this  process,  the 
members  work  by  themselves  at  a  time  and  place  of  their  choosing. 

2.  Assessment  Group  2 

The  District  of  Columbia  Strategic  Planning  Committee,  Detection  Research  and 
Development  Subcommittee  is  tasked  with  evaluating  and  integrating  technologies  for 
chemical,  biological,  radiation,  nuclear  and  explosives  detection  with  everyday  polices  as 
defined  in  the  District  of  Columbia  Strategic  Plan.  This  committee  must  integrate  needs 
of  the  district  government  with  those  of  federal,  state  and  private  sector  partners  in  order 
to  provide  layered  detection  capabilities  within  the  National  Capitol  Region. 

Members  of  this  committee  are  subject-matter  experts  in  detection  technologies  or 
response  policies  of  their  respective  agencies.  The  members  routinely  collaborate  with 
partners  outside  of  their  home  agencies  in  furtherance  of  the  goals  and  objectives  of  the 
committee. 

Group  two  members  will  be  employed  in  a  process  that  studies  the  results  of 
members  completing  the  assessment  on  their  own,  then  coming  together  to  reach 
consensus  on  the  results.  In  theory,  this  method  will  shorten  the  time  needed  to  work 
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together  while  still  allowing  members  collaborative  opportunity  by  virtue  of  the 
consensus  process.  This  method  may  reduce  the  time  impact  of  the  assessment  process 
by  increasing  flexibility  in  when  and  how  the  survey  is  completed.  This  method  is 
similar  to  the  read  ahead  method  used  in  meeting  environments  to  assure  that  members 
are  familiar  with  material  to  be  covered  and,  as  a  result,  reduce  the  amount  of  time  spent 
in  a  meeting. 


3.  Assessment  Group  3 

The  Metropolitan  Washington  Council  of  Governments,  Fire  Chiefs  Hazardous 
Materials  Subcommittee  mission  is  to  coordinate  the  policies  and  procedures  for 
hazardous  materials  response  teams  in  the  National  Capitol  Region.  This  committee  has 
been  in  existence  since  1982  and  has  developed  and  implemented  regional  policies  for 
Radiation  Response,  Rapid  Response  and  overseen  regional  training  programs  in 
Weapons  of  Mass  Destruction  Training  for  First  Responders.  Federal,  state  and  local 
governments  are  represented  on  the  committee  as  are  Department  of  Defense  assets,  the 
private  sector,  our  neighboring  Urban  Area  Security  Initiative  region  and  many 
disciplines. 

Group  3  will  be  used  to  examine  the  consensus  method  for  completing  the 
assessment.  In  this  process,  the  members  of  the  Hazardous  Materials  Subcommittee  will 
convene  with  the  purpose  of  completing  the  assessment  tool.  For  each  measure,  the 
members  will  come  to  consensus  on  the  status  of  the  measure.  This  result  will  be 
recorded  as  the  status  of  this  capability. 

Consensus  is  the  method  suggested  by  Pilot  Capabilities  Assessment  Project 
Working  Group  as  preferred  to  complete  the  assessment  tool.  Consensus  is  the  preferred 
method  because  it  is  believed  the  process  will  foster  collaborative  relationships  that  may 
extend  beyond  completion  of  the  assessment  tool.  The  working  group  believes  that  the 
collaborative  regional  relationships  are  important  to  building  improved  capability  with 
the  NPG.25 

25  Department  of  Homeland  Security,  Pilot  Capability  Assessment  Plan  Working  Document  (DHS: 
Washington,  DC,  March  3,  2006),  11 
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4. 


Assessment  Group  4 


This  study  was  originally  designed  with  three  groups.  During  the  assessment 
process  with  Assessment  Group  1,  a  parallel  assessment  group  was  initiated  by  group 
members.  Because  the  research  question  involves  identifying  both  the  benefits  of 
collaborative  assessment  process  and  future  collaborative  opportunities  arising  from  the 
assessment  process,  I  decided  to  include  this  newly  formed  group  as  Assessment  Group 
4.  The  details  of  how  this  group  was  formed  are  discussed  in  Chapter  IV,  Analysis, 
Conclusion. 

This  group  is  composed  of  members  of  the  Council  of  Governments  Medical 
Surge  Hospitals  and  Health  Subcommittee.  This  group  represents  local  public  health 
officials,  private  sector  representatives,  emergency  managers,  and  first  responders.  In  the 
case  study,  this  group  employed  the  consensus  method  for  completing  the  assessment.  In 
this  process,  the  members  of  the  group  convened  with  the  purpose  of  completing  the 
assessment  tool.  For  each  measure,  the  members  came  to  consensus  on  the  status  of  the 
measure.  Their  results  were  recorded  as  the  status  of  this  capability. 

D.  INTERVIEWS 

The  product  of  each  group  —  a  completed  capability  assessment  perfonned  with 
the  Self  Assessment  Tool  —  provides  the  data  used  in  this  analysis.  The  first  step  in  the 
analysis  is  to  discover  if  the  assessment  tool  can  be  used  to  facilitate  data  measurements 
that  are  reflective  of  the  capabilities  being  measured.  To  reach  this  goal,  results  of  the 
group  products  will  be  compared  to  each  other  in  an  attempt  to  discover  similarities  and 
differences.  This  comparison  of  data  sets  will  be  used  to  identify  any  effect  of  the 
different  processes  in  performing  the  assessment. 

Interviews  with  focus  group  members  will  be  structured  and  include  questions 
pertaining  to  how  the  process  affected  how  the  assessment  data  would  be  used  and, 
outside  of  the  process,  if  there  are  other  factors  that  inhibit  participation  in  the  assessment 
process.  A  comparison  of  the  results  of  the  assessments  and  the  results  of  the  interviews 
will  provide  the  data  for  this  thesis. 
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From  this  analysis,  questions  for  each  of  the  participants  were  developed.  The 
results  of  the  interviews  were  used  to  confirm  observations  about  collaboration  and 
inform  policy  recommendations  about  which  of  the  assessment  processes  holds  the  most 
promise  for  providing  a  reliable  capability  assessment  and  creating  additional  benefits. 

E.  LIMITS  OF  THE  EMPLOYED  METHODOLOGY 

There  are  limits  to  how  the  results  of  these  case  studies  should  be  compared  and 
interpreted.  The  methodology  is  designed  to  examine  the  process  used  in  the  assessment. 
This  study  should  have  been  performed  using  the  multiple  randomly-assigned  assessment 
groups  to  perform  the  assessment  utilizing  a  different  process.  This  was  not  done. 
Instead,  the  assessment  groups  were  composed  of  existing  groups.  The  decision  to 
perform  the  case  studies  in  this  manner  was  made  because  the  DHS  proposed 
methodology  performs  the  assessment  one  time,  by  assessment  groups  with  similar 
compositions  to  the  study  assessment  groups. 

The  use  of  existing  groups  as  assessment  teams  also  serves  to  limit  how  data  from 
this  study  can  be  interpreted.  A  more  reliable  method  would  be  to  utilize  randomly 
selected  subject-matter  experts  to  perform  the  assessments.  In  reality,  the  small  size  of 
the  subject-matter  expert  community,  and  the  heavy  time  demands  placed  upon  them 
because  of  their  expertise,  eliminated  this  option.  This  decision  is  also  consistent  with 
how  the  region  has  dealt  with  performing  capabilities  assessments  in  the  past,  assigning 
the  assessment  responsibility  to  an  existing  group. 

The  use  of  existing  groups  does  not  control  for  selection  bias,  and  as  a  result  some 
of  the  suggest  findings  may  have  alternative  explanations.  The  alternative  explanations 
may  come  from  group  composition  and  not  the  change  in  process.  This  limit  will  be 
especially  important  to  consider  when  comparing  group  results. 

These  methodology  compromises  ultimately  limit  the  size  and  scope  of  the  study. 
As  a  result,  the  data  produced  by  the  case  studies  should  be  recognized  for  its 
weaknesses,  and  the  analysis  of  the  data  collected  during  the  study  can  only  be 
recognized  as  suggestive  or  exploratory  findings. 
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IV.  ANALYSIS 


A.  ANALYSIS  INTRODUCTION 

This  thesis  examined  a  proposed  methodology  for  perfonning  the  National 
Capabilities  Assessment  required  by  HSPD  8.  Within  the  proposed  methodology,  a 
question  of  which  process  would  be  most  appropriate  to  detennine  the  status  of 
capabilities  became  pertinent.  The  study  within  this  thesis  was  developed  to  examine 
potential  assessment  processes  to  determine  if  a  proposed  approach  had  benefits  over 
other  possible  approaches. 

This  analysis  is  limited  to  the  small  set  of  capabilities  surveyed  in  the  comparative 
case  study.  The  study  is  performed  within  the  guidelines  of  the  proposed  methodology. 
The  study  produced  a  data  set  that  is  suggestive  of  results  that  may  be  found  in  other 
Urban  Area  Security  Initiative  regions,  but  cannot  claim  to  represent  all  of  homeland 
security.  The  data  set  represents  many  of  the  homeland  security  disciplines,  but  does  not 
provide  multiple  viewpoints  from  some  of  the  support  disciplines.  Interviews  with  survey 
participants  were  used  to  clarify  the  existence  of  trends  in  the  data  and  confirm 
observations  about  collaboration,  validity  and  reliability. 

B.  CAPABILITY  ASSESSMENT  COMPARATIVE  CASE  STUDY 

The  Pilot  Capability  Assessment  Process  is  designed  to  employ  a  group  of 
subject-matter  experts  to  assess  the  status  of  the  various  capabilities  defined  in  the  TCL. 
The  assessment  process  is  guided  through  the  use  a  self-assessment  tool.  This  tool 
employs  a  set  of  measures  for  each  capability.  Using  the  scoring  scale  in  Figure  1,  the 
measures  are  rated  by  the  assessment  group  and  collectively  represent  the  status  of  the 
capability. 
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Figure  1. 


Pilot  Capability  Self  Assessment  Tool  Scoring  Scale.26 


Label 

Explanation 

Scale  Value 

No  Progress 

A  score  of  0  indicates  that,  while  this  aspect  of  [name  the 
capability]  is  applicable  to  the  region,  no  progress  has 
been  made  towards  achieving  the  identified  objective.  This 
may  be  because  there  has  been  no  activity  in  this  area,  or 
because  insurmountable  barriers  exist. 

0 

Limited 

Progress 

Regions  should  score  themselves  at  the  lower  to  middle 
part  of  this  range  if  preliminary  efforts  have  been  initiated. 
Needs  related  to  this  objective  have  been  recognized,  and 
the  region  is  just  beginning  to  identify  requirements  in  this 
area.  Few  if  any  steps  have  been  taken  so  far. 

Regions  should  score  themselves  at  the  middle  to  upper 
part  of  this  range  if  the  region  has  analyzed  needs  and 
understands  requirements,  and  has  taken  specific  steps 
toward  achieving  the  objective.  These  steps  may  include 
initial  plans  to  develop  this  aspect  of  [name  the  capability], 
allocation  of  resources,  and  identification  personnel 
responsible  for  achievement  of  the  objective. 

1 

2 

3 

Moderate 

Progress 

Regions  should  score  themselves  at  the  lower  to  middle 
part  of  this  range  if  significant  efforts  are  under  way  but  the 
objective  has  not  yet  been  fulfilled.  Important  gaps  remain 
and/or  challenges  that  could  potentially  undermine 
achievement  exist  and  have  not  yet  been  resolved. 

Regions  should  score  themselves  at  the  middle  to  upper 
part  of  this  range  if  significant  efforts  are  under  way  and 
specific  examples  of  progress  in  this  area  can  be  identified. 
Strategies  for  closing  gaps  and  overcoming  barriers  to 
success  are  being  developed  and  initiated. 

4 

5 

6 

Substantial 

Progress 

Regions  should  score  themselves  at  the  lower  to  middle 
part  of  this  range  if  efforts  to  achieve  this  objective  are 
established  and  stable.  Some  weaknesses  or  barriers  that 
prevent  success  persist,  but  strategies  to  resolve  them  are 
documented  and  well  under  way. 

Regions  should  score  themselves  at  the  middle  to  upper 
part  of  this  range  if  efforts  in  this  area  are  mature.  Few 
gaps  or  barriers  to  success  remain.  None  are  significant. 
Evidence  documenting  this  level  of  progress  is  readily 
available.  Evidence  may  include  After  Action  Reports  from 
exercises  or  events  where  this  aspect  of  capability  was 
demonstrated. 

7 

8 

9 

Objective 

Achieved 

A  score  of  10  indicates  that  the  region  has  fully  achieved 
this  objective  with  regard  to  [name  the  capability].  All 
barriers  to  success  have  been  overcome.  Strengths  are 
robust  and  likely  to  be  sustained.  Evidence  is  readily 
available  attesting  to  this  level  of  achievement. 

10 

Not  Applicable 

The  aspect  of  [name  the  capability]  that  this  question 
describes  is  not  applicable  to  the  region. 

N/A 

26  Self  Assessment  Tool  Version  2.0,  Pilot  Assessment  Working  Group  (DHS,  FEMS,  NPD: 
Washington,  DC:  January  29,  2007),  Tear  Out  Sheet  Tab. 
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The  National  Capital  Region  employs  a  homeland  security  strategy  based  upon 
the  National  Response  Plan.27  Under  this  strategy,  the  Target  Capabilities  in  the  NPG  are 
assigned  to  Emergency  Support  Functions  for  implementation.  The  two  Target 
Capabilities  for  assessment  were  selected  to  assure  that  data  came  from  a  cross-section  of 
emergency  support  functions  and  from  target  capabilities  with  different  emergency 
support  function  responsibility. 

Health  is  the  designated  lead  for  Emergency  Support  Function  #  8,  Health,  Mental 
Health  and  Medical  Services.  This  group  has  been  tasked  to  lead  the  implementation  of 
the  Mass  Surge  capabilities.  The  effort  is  supported  by  the  emergency  support  functions 
for  #  1,  Transportation;  #  4,  Firefighting,  #  5,  Communications  and  Planning;  #  13,  Law 
Enforcement;  and  #15,  Donations  and  Volunteer  Management. 

The  Fire  Department  has  been  designated  as  the  lead  agency  for  Emergency 
Support  Function  #  10,  Hazardous  Materials.  This  group  has  been  assigned  the 
responsibility  to  implement  the  Hazardous  Materials/WMD  Response  and 
Decontamination  Target  Capability.  This  effort  is  supported  by  #  1,  Transportation;  #  5, 
Communications  and  Planning;  #  8,  Health,  Mental  Health,  and  Medical  Services;  #  13, 
Law  Enforcement;  and  #15,  Donations  and  Volunteer  Management. 

The  selection  of  these  groups  assures  a  similar  support  base  across  capabilities  to 
be  measured  and  will  provide  continuity  in  the  assessment  process.  The  members 
participating  in  the  assessment  are  people  who  work  together  on  many  of  the  same  issues 
being  assessed  as  a  regular  course  of  their  employment. 

1.  Assessment  Tool 

A  Pilot  Capability  Assessment  Working  Group  created  the  Self  Assessment  Tool 
to  assess  capabilities.  This  goal  of  this  tool  is  to  allow  local  preparedness  officials  to 
participate  in  a  process  of  “capabilities  assessment  [that]  produces  valuable  and  relevant 
preparedness  infonnation  [and]  supports  a  regional  preparedness  approach  to  planning 


27  The  National  Capital  Response  Plan  is  formulated  but  not  ratified,  but  working  drafts  are  in  place. 
Planning  and  Response  functions  are  routinely  using  the  draft  form  of  the  plan. 
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and  investment  efforts.”28  This  study  was  not  designed  to  examine  the  tool  directly,  but 
will  examine  potential  inconsistencies  in  the  tool  or  measures  suggested  by  the  data. 

The  tool  uses  a  series  of  measures  that  are  designed  to  assess  the  progress  of  a 
region  in  implementation  of  the  NPG.  Selecting  portions  of  the  tool  that  provide  the 
opportunity  to  measure  the  response  of  multiple  disciplines  across  two  dissimilar 
capabilities  allows  for  an  evaluation  of  the  tool.  For  this  survey,  the  Target  Capabilities 
to  be  assessed  are  Mass  Surge  and  Hazardous  Materials/WMD  Response  and 
Decontamination. 

The  tool  is  designed  for  a  small  group  of  subject-matter  experts  to  evaluate  a  set 
of  cross-capability  measures  about  collaboration  and  a  set  of  specific  measures  about  a 
defined  capability.  The  assessment  process  is  guided  through  the  use  a  Self  Assessment 
Tool.  This  tool  employs  a  set  of  measures  for  each  capability.  Using  the  scale  in  Figure 
1,  the  measures  are  rated  by  the  assessment  group  and  collectively  represent  the  status  of 
the  capability. 

The  scoring  scale  is  comprised  of  a  subjective  numeric  scale  tied  to  labels  and 
explanations  of  progress  that  are  used  to  represent  progress  toward  meeting  a  specified 
measure.  The  number  scale  is  anchored  on  either  end  by  the  number  “0,”  indicating  no 
progress  toward  meeting  the  measure  and  the  number  “10,”  indicating  the  measure  is 
completely  satisfied.  The  number  scale  is  further  divided  by  explanations  that  create  sets 
of  three  numbers  into  three  labeled  categories  of  effort. 

To  develop  the  tool,  the  Pilot  Capabilities  Working  Group  refined  characteristics 
of  a  good  assessment  tool  for  data  collection  using  after-action  reports  on  pilot 
assessment  programs  conducted  in  Florida  and  Colorado.  They  declared  that  survey 
measures  should  have  the  following  characteristics: 

•  The  measures  should  be  a  single-stage  question,  not  a  multipart  question. 

•  The  measures  should  provide  enough  detail  and  information  so  the 
measure  has  universal  meaning  to  respondents. 

28  Seamus  Mooney,  “Pilot  Capabilities  Assessment  Update  &  Methodology  Discussion,”  Pilot 
Capability  Assessment  Working  Group  (FEMA,  NPD:  Washington,  DC,  August  15,  2007),  4. 
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•  Training  and  resources  measures  should  be  separate  from  plans  and 
equipment. 

•  Equipment  measures  should  address  all  aspects  (procurement,  training  and 
maintenance)  of  the  use  of  the  equipment.29 

2.  Precision 

Because  the  label  scale  is  grouped  by  threes,  it  can  be  inferred  that  the  accepted 
level  of  precision  through  use  of  the  tool  is  a  range  of  three  points.  Therefore,  any 
measures  of  progress  by  assessment  groups  that  are  within  three  points  of  each  other 
constitute  agreement  on  the  rating  given  to  a  measure. 

The  data  in  this  analysis  was  obtained  from  an  experiment  that  employed  three 
different  processes  to  complete  a  self  assessment.  Each  of  the  self  assessments  utilize  the 
same  measures  to  rate  the  status  of  the  Mass  Surge  and  Hazardous  Materials  and 
Weapons  of  Mass  Destruction  Response  and  Decontamination  capabilities.  The 
assessments  were  performed  by  local  subject-matter  experts;  the  same  people  who  have 
performed  in  assessment  roles  in  the  past  and  have  responsibilities  for  implementing 
programs  in  the  capability  areas  being  assessed. 

The  success  of  this  comparative  case  study  and  the  capability  assessment  assumes 
that  the  assembled  subject-matter  experts  have  the  requisite  knowledge  needed  to  assess 
the  measures,  and  that  collaboration  among  these  subject-matter  experts  will  produce  a 
valid  and  reliable  convergence  of  ratings. 


29  Department  of  Homeland  Security,  Pilot  Capabilities  Assessment  Project,  Meeting  Notes,  October 
25,  2006  (Department  of  Homeland  Security,  Office  of  Grants  and  Training):  2. 
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Figure  2.  Validity  and  Reliability  Examples.30 


Figure  2  represents  the  possible  outcome  scenarios  from  our  assessment  groups. 
The  red  dot  in  the  center  represents  the  undetermined  status  of  the  measure,  and  the  blue 
dots  represent  assessments  of  the  measure.  In  this  assessment  process,  a  reliable  and 
valid  outcome  is  multiple  subject-matter  experts  providing  similar  ratings  for  a  measure. 

C.  RELIABILITY 

In  the  comparative  case  study,  four  separate  groups  of  subject-matter  experts 
completed  the  assessment  utilizing  the  measures  in  the  Self  Assessment  Tool  to  rate  the 
progress  toward  implementation  of  the  Hazardous  Materials  /  WMD  Response  and 
Decontamination  Capability.  Reliable  data  indicating  capability  status  from  this  survey 
would  be  indicated  by  measures  with  scores  grouped  around  a  range  of  three  points  on 
the  scale;  or  grouped  together  as  indicated  in  Figure  2,  “Reliable  and  Valid.”  The  range 
of  three  points  was  earlier  described  as  the  precision  of  the  Pilot  Self  Assessment  Tool.  If 
the  range  exceeded  three  points  across  a  specific  measure,  we  could  infer  that  the  results 
for  this  assessment  process  are  unreliable.  Because  this  thesis  is  examining  the  effect  of 
process  when  using  the  tool,  this  finding  would  suggest  further  study  of  the  process  and 
the  assessment  groups. 


30  William  M.  K.  Trochim,  “Reliability  &  Validity,”  Research  Methods  Knowledge  Base”  (Web 
Center  for  Social  Research  Methods),  October  10,  2006, 
http://wwsocialresearchmethods.net/kb/relandval.php,  assessed  July  17,  2007. 
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Figure  2  also  shows  an  example;  “Reliable,  not  Valid,”  that  can  occur  when  the 
range  of  ratings  is  within  a  three-point  range  but  does  not  adequately  relate  to  the 
underlying  capability  status.  Participants  may  agree,  but  in  this  example  they  may  be 
agreeing  on  the  basis  of  wrong  information  about  the  capability.  An  examination  of  any 
grouping  warrants  examination  to  determine  validity. 


— Group  1  — ■—  Group  2  Group  3  Group  4 


Figure  3.  Self  Assessment  Group  Ratings,  Hazardous  Materials  /  WMD  Response  and 

Decontamination  Capability. 


The  chart  in  Figure  3  represents  a  comparison  of  the  Self  Assessment  Group 
ratings  for  the  fifteen  measures  in  the  Hazardous  Materials  /  WMD  Response  and 
Decontamination  capability  assessment.  This  chart  clearly  shows  that  only  measures  3, 
7,  10  and  15  produce  a  grouping  of  results  within  the  desired  three-point  range  for 
reliability.  The  remaining  assessment  results  exceed  the  three-point  range.  This 
observation  suggests  the  need  for  further  study  of  the  effect  of  the  assessment  process  on 
the  validity  of  the  results. 


29 


10.0000 


9.0000 

8.0000 

7.0000 

6.0000 

5.0000 

4.0000 

3.0000 

2.0000 

1 .0000 

0.0000  4 


*.<2r  vgr  v 2r  vgr  kZt  *J2r  'Zr  w5  <7v>  <2?  17?  w5  <??  it?  w5  i??  <7?  w5  w  17?  it?5 


-Group  1 


-Group  2 


Group  3 


Group  4 


Figure  4.  Self  Assessment  Group  Ratings,  Mass  Surge  Capability. 


The  chart  in  Figure  4  represents  Self  Assessment  Group  ratings  for  measures  in 
the  Mass  Surge  capability  assessment.  In  this  case,  only  measures  1 1  and  12  exceed  the 
three-point  range  for  accuracy.  The  finding  from  this  comparison  suggests  that  the 
different  assessment  processes  produced  reliable  groupings  of  data.  Further  examination 
of  this  data  will  be  completed  to  examine  validity. 

An  examination  of  the  reliability  of  the  assessment  processes,  as  measured  by 
group  outputs,  suggests  that  there  could  be  problems  in  the  Hazardous  Materials  /  WMD 
Response  and  Decontamination  assessment  and,  to  a  lesser  degree,  in  the  Mass  Surge 
assessment.  A  finding  that  the  assessment  process  failed  to  produce  reliable  results 
suggests  the  need  for  further  examination  of  the  methodology.  If  true,  unreliability  is  an 
important  finding  about  the  capability  assessment  process. 

The  findings  in  this  section  of  the  analysis  explore  problems  with  data  reliability. 

These  findings  also  suggest  a  need  to  examine  the  validity  of  the  data  collected.  The 
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examination  of  validity  will  be  prefaced  by  a  discussion  on  collaboration,  because  the 
extent  and  methods  of  collaboration  provides  evidence  on  the  validity  of  the  assessment 
results. 

D.  COLLABORATION  AND  CONVERGENCE 

In  addition  to  providing  data  about  capability  status,  the  expected  outcomes  of  the 
assessment  process  are  that  it  will  both  educate  and  encourage  collaboration.31  This  study 
provided  an  opportunity  to  explore  the  educational  and  collaborative  aspects  of  the 
assessment  process  by  examining  the  data  and  interviewing  participants  to  understand 
how  members  came  to  agree  on  ratings  for  measures.  The  expected  result  from  the  study 
is  that  members  of  the  assessment  groups  would  come  to  the  process  with  a  certain  level 
of  understanding  of  the  capability  status.  Also,  as  a  result  of  collaboration  with  other 
group  members,  they  would  gain  additional  knowledge  about  the  capability  status,  thus 
converging  on  the  actual  capability  status. 

An  observation  of  Assessment  Group  2  allows  an  understanding  of  how 
individual  assessment  group  members  came  to  a  consensus  for  group  ratings.  These 
observations  show  how  the  assessment  process  produced  a  final  rating.  This  final  rating 
should  represent  a  convergence  of  the  individual  group  member’s  knowledge  about  the 
capability  status.  If  a  convergence  in  the  ratings  does  not  occur,  problems  related  to 
group  think  and  assessment  design  need  to  be  considered. 

Observations  of  Assessment  Group  2  were  used  to  examine  the  collaborative 
effort  employed  to  meet  the  goal  of  performing  a  multi-disciplined  assessment. 
Interviews  with  individual  group  members  were  used  to  detennine  if  convergence 
occurred  in  reaching  the  final  rating,  or  if  agreement  was  influenced  by  other  behaviors. 


31  Department  of  Homeland  Security,  "Pilot  Capability’  Assessment  Plan  Working  Document”  (DHS: 
Washington,  DC,  March  3,  2006),  2. 
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Measure12  Final  Determined  Capability  Status 


Figure  5.  Assessment  Group  2,  Haz  Mat  /  WMD  Response  and  Decontamination, 
Measure  12,  Individual  Assessment  versus  Group  Assessment. 


Figure  5  demonstrates  a  comparison  of  the  level  of  progress  ratings  from  the 
members  of  Assessment  Group  2  for  Measure  12; 

Our  region  is  able  to  ensure  the  safety  and  security  at  the  scene  and 
support  facilities  it  intends  to  use  for  WMD/Hazardous  Materials 
Response  and  Decontamination.  (That  is,  can  your  region  determine  and 
ensure  the  structural  integrity,  capability,  and  physical  security  of  its 
HazMat  Management  facilities?)32 

The  chart  demonstrates  diversity  of  individual  opinion  about  the  status  of 
Measure  12.  The  ratings  range  of  eight  points  exceeds  the  three-point  precision  of  the 
tool  and  would  cause  the  assessment  result  to  be  considered  unreliable  if  those  views 
remained  as  the  final  opinion  of  the  assessment  group.  The  chart  also  demonstrates  a 


32  Self  Assessment  Tool  Version  2.0,  Pilot  Assessment  Working  Group  (DHS,  FEMS,  NPD: 
Washington,  DC:  Hazardous  Materials  /  WMD  Response  and  Decontamination  Tab,  January  29,  2007. 
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comparison  between  individual  determination  of  level  of  progress  and  final  group 
determination  of  level  of  progress  for  the  measure.  Finally,  the  chart  also  shows  where 
a  member  of  the  health  discipline  did  not  provide  a  rating  for  the  measure  because  of 
uncertainty  as  to  the  level  of  progress.33 


Figure  6.  Assessment  Group  2,  Haz  Mat  /  WMD  Response  and  Decontamination, 
Measure  6,  Individual  Assessment  versus  Group  Assessment. 


Figure  6  demonstrates  a  comparison  of  level  of  progress  ratings  from  members  of 
Assessment  Group  2  for  Measure  6: 


33  Instructions  to  assessment  team  members  completing  the  survey  were  to  complete  it  to  complete  the 
survey  to  the  best  of  their  ability,  follow  up  questions  from  members  resulted  in  assessment  team  members 
being  directed  to  not  rank  measures  they  did  not  “know  the  answer  to.” 
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Our  region’s  plans  for  WMD/Hazardous  Materials  Response  and 
Decontamination  address  substance  identification  equipment  (e.g.,  bases, 
vapors,  liquids,  solids,  biologicals  like  white  powder).34 

The  chart  demonstrates  diversity  of  individual  opinion  about  the  status  of 
Measure  6.  The  ratings  range  of  five  points  exceeds  the  three-point  precision  of  the  tool 
and  would  cause  the  assessment  result  to  be  considered  unreliable  if  those  views 
remained  as  the  final  opinion  of  the  assessment  group.  The  chart  also  demonstrates  a 
comparison  between  individual  determination  of  level  of  progress  and  final  group 
determination  of  level  of  progress  for  the  measure.  Finally,  the  chart  also  shows  where 
a  member  of  the  health  disciplines  did  not  provide  a  rating  for  the  measure  because  of 
uncertainty  as  to  the  level  of  progress. 


Measure14  — I — Final  Determined  Capability  Staus 


Figure  7.  Assessment  Group  2,  Mass  Surge,  Measure  14,  Individual  Assessment  versus 

Group  Assessment. 


34  Self  Assessment  Tool  Version  2.0,  Pilot  Assessment  Working  Group  (DHS,  FEMS,  NPD: 
Washington,  DC:  Hazardous  Materials  /  WMD  Response  and  Decontamination  Tab,  January  29,  2007. 
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Figure  7  demonstrates  a  comparison  of  the  level  of  progress  ratings  from  the 
members  of  Assessment  Group  2  for  Measure  14: 

Our  region  maintains  databases  that  track  the  status  of  resources  (e.g., 

medications,  medical  professionals)  available  to  support  Medical  Surge.35 

The  chart  demonstrates  diversity  of  individual  opinion  about  status  of  Measure 
14.  The  ratings  range  of  six  points  exceeds  the  three-point  precision  of  the  tool  and  would 
cause  the  assessment  rating  to  be  considered  unreliable  if  those  views  remained  as  the 
final  opinion  of  the  assessment  Group.  The  chart  also  demonstrates  a  comparison 
between  individual  detennination  of  level  of  progress  and  final  group  determination  of 
level  of  progress  for  the  measure.  Finally,  the  chart  also  shows  where  members  of  the 
police  and  fire  disciplines  did  not  rate  the  measure  because  of  uncertainty  about  the  level 
of  progress. 

E.  VALIDITY 

The  foundation  of  this  assessment  methodology  rests  on  the  ability  of  subject- 
matter  experts  to  reach  an  accurate  conclusion  on  the  status  of  a  capability.  In  the 
proposed  methodology,  the  subject-matter  experts  are  selected  by  state  and  local 
governments.  Within  local  and  state  government,  the  functional  area  being  measured  is  a 
niche;  at  this  time  much  of  the  detailed  knowledge  of  homeland  security  activities  is  with 
a  small  cadre  of  personnel.  Because  the  community  is  small,  care  must  be  taken  to 
ensure  that  assessment  groups  are  representative  of  a  cross-section  of  this  community  in 
order  to  ensure  diversity  of  opinion.  Ultimately,  in  this  study,  validity  is  measured  by 
agreement  of  the  groups  and  members  as  measured  in  assessment  ratings. 

The  charts  presented  represent  the  wide  range  of  opinions  on  the  level  of  progress 
being  made  towards  the  capabilities  held  by  individual  members  of  the  assessment  group, 
including  examples  where  members  felt  they  did  not  know  the  level  of  progress  for  a 
measure.  If  the  average  rating  reflects  the  member’s  “informed”  opinion  about  the  status 

35  Self  Assessment  Tool  Version  2.0,  Pilot  Assessment  Working  Group  (DHS,  FEMS,  NPD: 
Washington,  DC:  Mass  Surge  Tab,  January  29,  2007. 
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of  the  capability,  then  it  is  necessary  to  understand  how  the  assessment  groups  reached 
consensus  on  the  rating  for  a  measure.  This  point  is  critical  to  understanding  the  validity 
of  each  measure  and,  through  each  measure,  the  entire  process.  The  design  of  the  tool 
expects  consensus  to  create  validity  as  assessment  group  members  employ  their  expertise 
to  validate  the  assessment  results. 

1.  Validity  Assessment  Group  2 

When  Assessment  Group  2  met  to  reach  consensus  on  the  assigned  rating  for  each 
measure,  group  members  accepted  the  aggregated  result  each  time.  In  the  consensus 
process,  there  were  discussions  about  the  level  of  progress,  and  pointed  questions  asked 
between  disciplines.  These  discussions,  however  failed  to  move  the  group  from  the 
aggregated  rating.  The  questions  asked  between  members  of  the  group  were  focused  on 
learning  about  the  status  of  the  capability,  as  opposed  to  efforts  to  have  other  members 
change  their  opinion  about  a  capability  status. 

For  example,  members  of  the  police  discipline  inquired  with  the  health  discipline 
about  integration  of  security  expectations  in  the  measures.  These  discussions  exhibited 
the  characteristics  of  free  flow  of  information  and  discovery  that  are  the  elements  of 
collaborative  effort.36 

Interviews  were  utilized  to  examine  the  occurrence  of  convergence  in  the 
assessment  ratings,  specifically  to  examine  why  members  agreed  to  accept  ratings  that 
differed  greatly  from  their  own.  Interviews  with  group  members  found  that  members 
routinely  accepted  the  average  rating  because  they  were  uncertain  of  their  original  rating. 
Evidence  of  the  lack  of  confidence  in  the  original  rating  or  the  final  rating  is 
demonstrated  in  the  interview  process.  Reasons  given  for  accepting  the  aggregation 
included: 


36  Hocevar,  Susan,  Gail  Thomas  and  Erik  Jansen,  “Building  Collaborative  Capacity:  An  Innovative 
Strategy  for  Homeland  Security  Preparedness,”  Innovation  Through  Collaboration,  Advances  in 
Interdisciplinary  Studies  of  Work  Teams  12  (Elseier  Ltd:  2005),  264. 
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•  Looking  at  the  scores  now,  I  went  too  high.37 

•  I  gave  an  honest  answer,  not  sure  everyone  else  gave  honest  answer.38 

•  I  am  an  optimist,  but  the  group  might  be  producing  a  more  accurate  rating.39 

It  is  a  significant  finding  that  group  members  accepted  the  mean  ratings  without  a 
single  change  every  time.  They  accepted  the  changes  despite  evidence  of  a  wide  range  of 
opinions  as  to  what  the  correct  rating  could  have  been.  This  finding  is  not  an  example  of 
convergence  in  the  sense  that  convergence  equals  knowledge  gained.  It  is  a  finding  that 
members  changed  their  assessments  because  they  lacked  confidence  in  their  knowledge. 
This  finding  is  true  for  both  capabilities  assessed.  This  finding  suggests  exploring  the 
assumption  that  subject-matter  experts  have  the  expertise  to  conduct  reliable  and  valid 
capability  assessments. 

2.  Validity  of  Assessment  Group  1 

To  support  the  suggestion  of  validity  problems  from  the  analysis  of  data  from 
Assessment  Group  2,  an  examination  of  the  data  collected  from  Assessment  Group  1  was 
conducted.  This  data  is  important  because  it  also  suggests  that  the  selected  subject-matter 
experts  lack  the  knowledge  to  assess  capabilities  against  the  proposed  measures. 

Figure  8  demonstrates  individual  ratings  from  Assessment  Group  1  for  Hazardous 
Materials  /  WMD  Response  and  Decontamination,  Measure  15,  “To  what  extent  has  your 
region  exercised  its  ability  to  implement  WMD/Hazardous  Materials  Response  and 
Decontamination  procedures  (e.g.,  exercised  response  to  pre-identified  hazards, 
decontaminated  victims).”40  The  chart  demonstrates  diversity  of  individual  opinion 
about  the  status  of  Measure  15.  The  eight-point  ratings  range  exceeds  the  three-point 
precision  of  the  tool.  The  chart  also  demonstrates  a  comparison  between  individual 

37  Interview  of  Participant  5  by  author,  Washington,  DC,  June  6,  2007. 

38  Interview  of  Participant  6  by  author,  Washington,  DC,  June  5,  2007. 

39  Interview  of  Participant  7  by  author,  Washington,  DC,  June  5,  2007. 

40  Self  Assessment  Tool  Version  2.0,  Pilot  Assessment  Working  Group  (DHS,  FEMS,  NPD: 
Washington,  DC:  Hazardous  Materials  /  WMD  Response  and  Decontamination  Tab,  January  29,  2007. 
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determination  of  level  of  progress  and  final  group  determination  of  level  of  progress. 
Finally,  the  chart  also  shows  where  a  member  of  the  fire  discipline  did  not  rate  the 
measure  because  of  uncertainty  as  to  the  level  of  progress. 


Measure15  Final  Assessed  Rating"] 


Figure  8.  Assessment  Group  1,  Hazardous  Materials  /  WMD  Response  and 

Decontamination,  Measure  15,  Individual  Assessment  versus  Group  Assessment. 


Figure  9  demonstrates  individual  ratings  from  Assessment  Group  1  for  Mass 
Surge,  Measure  3: 

Our  region’s  plans  address  the  use  of  existing  facilities  in  support  of 
Medical  Surge  (e.g.,  hospitals,  clinics,  extended  care  facilities).41 

The  chart  demonstrates  diversity  of  individual  opinion  as  to  level  of  progress 
towards  Measure  3.  The  seven-point  rating  range  exceeds  the  three-point  precision  of  the 
tool.  The  chart  also  demonstrates  a  comparison  between  individual  determination  of 

41  Self  Assessment  Tool  Version  2.0,  Pilot  Assessment  Working  Group,  (DHS,  FEMS,  NPD: 
Washington,  DC:  Mass  Surge  Tab  January  29,  2007. 
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level  of  progress  and  final  group  determination  of  level  of  progress.  Finally,  the  chart 
also  shows  where  two  members  of  the  emergency  management  discipline  did  not  rate  the 
measure  because  of  uncertainty  as  to  the  level  of  progress. 


Measure3 


-  Final  Determined  Capability  Status  | 


Figure  9.  Assessment  Group  1,  Mass  Surge,  Measure  3,  Individual  Assessment  versus 

Group  Assessment. 


At  the  conclusion  of  the  comparative  case  study,  members  of  Assessment  Group  1 
were  individually  presented  with  the  aggregated  results  and  queried  about  the  final 
ratings,  the  relationship  of  the  final  rating  to  their  rating  and  if  they  would  feel 
comfortable  using  the  results  of  the  assessment  in  planning  for  future  efforts. 

Assessment  Group  1  members  reacted  in  a  similar  manner  to  Assessment  Group  2 
members  in  accepting  the  final  aggregated  ratings  as  more  accurate  than  their  own 
opinion  on  the  status  of  the  capability.  Many  of  the  members  felt  they  might  use  this  data 
in  future  planning  efforts.  This  finding  is  troublesome,  because  it  presents  a  dilemma  of 
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inaccurate  information  about  the  status  of  a  capability  being  interjected  into  the 
preparedness  cycle  and  capability-based  planning  efforts  required  by  HSPD  8. 

The  data  suggests  that  disagreement  in  the  assessment  group  members’ 
confidence  in  their  individual  knowledge  was  exhibited  on  multiple  occasions.  One  of 
the  police  officials  who  completed  the  survey  felt  that  she  would  use  the  data  to 
determine  funding  or  project  priorities  between  the  assessed  capabilities.  She  did  say  that 
she  was  unsure  about  other  uses  of  the  assessment  results.42  A  member  of  the  emergency 
management  discipline,  who  did  not  provide  ratings  for  the  Mass  Surge  Capability,  felt 
that  the  results  would  need  further  study  to  determine  if  they  held  value.  This  member 
went  on  to  say  that  the  assessment  results  were  accurate  for  Mass  Surge,  based  on  his 
limited  knowledge,  and  he  was  confident  in  the  assessment  results  for  Hazardous 
Materials  /  WMD  Response  and  Decontamination. 

3.  Validity  Conclusions 

The  validity  of  this  assessment  methodology  is  based  on  the  subject-matter 
experts’  ability  to  reach  an  accurate  conclusion  about  the  status  of  a  capability.  As 
described  earlier,  the  community  of  experts  is  a  small  group  with  wide-ranging 
responsibilities  and  needs.  In  this  small,  diverse  group,  it  is  important  the  assessment 
process  represent  all  members.  Representing  all  members  will  ensure  the  diversity  of 
opinion  necessary  to  thoroughly  examine  and  accurately  assess  the  status  of  a  capability. 

There  is  a  limiting  factor  in  this  discussion  about  validity,  in  that  the  context  of 
this  thesis  validity  can  only  refer  to  the  agreement  among  the  assessment  teams.  Because 
of  the  aforementioned  limits  of  the  study  methodology,  the  findings  presented  in  this 
section  are  not  definitive,  but  explore  a  potential  problem  in  the  proposed  assessment 
process. 

The  four  groups  employed  in  this  study  utilizing  the  Self  Assessment  Tool  are 
representative  of  the  homeland  security  community.  Members  in  this  study  represent 
fourteen  different  disciplines,  federal,  state,  local  government  and  the  private  sector. 

42  Interview  Participant  3  by  author,  Washington  DC,  June  5,  2007. 
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More  importantly,  the  assessment  group  members  are  leaders  of  projects  or  committees 
implementing  the  capabilities  being  measured.  These  members  and  the  groups  they 
represent  are  the  right  people  to  perform  a  macro  level  capability  assessment  of  this 
specialized  knowledge  area  because  they  have  the  requisite  specialized  knowledge. 

Assessment  Group  1  was  composed  of  members  of  an  existing  committee  that  is 
working  to  implement  regional  mass  decontamination  standards.  Assessment  Group  2 
was  composed  of  an  existing  committee  that  meets  to  discuss  detection  technology  and 
implementation  programs,  plans  and  purchases  in  compliance  with  the  goals  of  the 
District  of  Columbia  Response  Plan.  Assessment  Group  3  was  composed  of  members  of 
a  regional  hazardous  materials  response  committee.  Assessment  Group  4  was  composed 
of  a  regional  hospital-centered  health  care  committee. 

These  committees  and  groups  are  responsible  for  updating  response  plans, 
submitting  grant  applications  and  implementing  plans  to  raise  their  community’s  level  of 
preparedness.  These  people  are  the  experts  in  their  fields.  The  finding  —  that  these 
people  lacked  confidence  in  the  measures  —  points  to  a  number  of  potential  problems. 
The  problems  could  be  their  knowledge,  the  measures,  or  the  methodology.  What  is 
suggested  by  this  study  is  that  the  capability  assessment  methodology  and  the  subject- 
matter  experts  are  misaligned.  The  result  of  this  misalignment  is  the  performance  of  an 
inaccurate  capability  assessment. 

An  earlier  suggestion  —  that  the  Mass  Surge  assessment  process  provided  reliable 
results  —  may  be  questioned  by  an  occurrence  during  the  Mass  Surge  capability 
assessment.  An  examination  of  Figure  8  shows  that  data  from  Assessment  Group  3  is  not 
included.  This  data  is  not  included  because  Assessment  Group  3  declined  to  complete 
the  Mass  Surge  capability  assessment.  Members  of  Group  3  asserted  they  did  not  have 
the  requisite  knowledge  to  accurately  complete  the  assessment.  They  felt  it  would  be 
better  to  abstain  from  the  process  than  to  provide  inaccurate  results.43  This  suggests  a 
significant  problem  related  to  validity.  The  suggestion  is  that  subject-matter  experts  were 
not  subject-matter  experts  for  the  purposes  of  the  capability  assessment  process. 

43  Minutes  of  the  Metropolitan  Washington  Council  of  Governments,  Fire  Chief  s  Hazardous 
Materials  Subcommittee  Meeting,  complied  by  Sean  Brooks,  April  13,  2007. 
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Non-response  to  a  question  or  category  of  questions  was  an  unexpected 
occurrence  in  this  study.  The  findings  here  suggest  a  pattern  that  questions  the  basis  for 
the  capability  assessment  methodology.  Subject-matter  experts  with  oversight  and 
planning  responsibilities  responded  to  the  survey  or  the  assessment  process  by 
apologizing  for  not  having  the  “correct”  answers  or  explaining  that  they  did  not  have  the 
requisite  knowledge  of  regional  efforts  to  rate  regional  progress  toward  the  goal.44  One 
fire  chief  apologized  for  not  collaborating  to  obtain  more  accurate  infonnation  to 
complete  the  survey;  he  also  felt  that  he  should  know  more  about  the  answers  to  the  Mass 
Surge  capability.45  This  admitted  lack  of  knowledge  is  consistent  with  the  wide  range  of 
individual  ratings  given  to  measures. 

Analysis  of  the  data  collected  from  the  assessment  process  suggests  further 
exploration  of  validity.  The  suggested  findings  related  to  validity  are  centered  on  the 
assumption  of  the  expertise  of  the  subject-matter  experts  selected  to  perfonn  the 
assessments.  The  problem  was  that  some  members  admitted  to  lacking  the  knowledge 
they  were  expected  to  have  to  complete  the  survey. 

These  suggested  findings  create  a  problem  with  data  collected.  In  the  assessment 
process,  the  status  of  the  capability  is  unknown;  if  the  assessment  ratings  are  determined 
to  be  invalid,  then  there  is  no  way  to  determine  the  status  of  the  capability.  The  next 
section,  on  collaboration  and  convergence,  will  be  used  to  examine  knowledge  levels  of 
the  subject-matter  experts  as  it  relates  to  this  process. 

F.  ASSESSMENT  DATA  BY  DISCIPLINE 

When  presented  with  the  results  of  the  survey,  members  of  one  discipline 
inevitably  wanted  to  know  how  the  other  disciplines  rated  a  measure  in  comparison  to  the 
rating  of  their  discipline.  This  comparative  case  study  provided  some  data  in  that  area 
and  suggests  some  trends  that  should  be  considered  in  future  methodology  development. 


44  Participant  1  email  to  author  April  3,  2007,  Participant  2  email  to  author,  March  27,  2007,  and 
Interview  of  Participant  3  by  author,  June  5,  2007. 

45  Participant  4  email  to  author,  March  30,  2007. 
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Figure  10.  Mass  Surge  Assessment  Ratings  by  Disciplines/Group. 


In  the  Mass  Surge  Capability,  the  health  discipline  is  designated  the  lead 
Emergency  Support  Function  for  the  development  and  implementation  of  plans  and 
programs  to  ensure  that  adequate  Mass  Surge  capability  is  available.  In  Figure  11,  the 
chart  presents  the  assessment  given  to  each  measure  according  to  discipline.  This  chart 
shows  that  members  of  the  health  discipline  tended  to  provide  higher  ratings  for  this 
capability  than  all  of  the  other  disciplines  except  police.  The  trending  shows  the  police 
and  health  discipline  generally  agreed  on  the  ratings  for  performance  measures. 

Figure  10  also  demonstrates  an  interesting  finding  about  group  versus  individual 
ratings.  The  health  discipline’s  aggregated  scores  are,  on  average,  far  above  Assessment 
Group  3’s  average.  This  is  significant  because  it  demonstrates  the  possible  differences 
between  an  aggregated  scoring  and  a  consensus  scoring  methodology.  Given  the 
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conflicted  data  presented  in  this  thesis,  it  is  impossible  to  determine  which  methodology 
provided  a  more  accurate  assessment  of  capabilities. 

The  emergency  management  discipline  is  a  bellwether  in  this  survey.  This 
discipline  has  overall  coordination  and  planning  responsibilities  that  cross  all  disciplines 
and  capabilities.  An  argument  can  be  made  that  if  the  higher  assessment  ratings  assigned 
by  the  health  discipline  were  accurate  the  emergency  management  discipline  would  also 
have  provided  a  rating  above  average,  instead  of  the  consistently  lower  rating  given. 

The  data  set  used  here  is  inconclusive  but  suggests  that  coordination  portions  of 
the  Mass  Surge  capability  plan  are  incomplete.  Three  members  of  the  emergency 
management  discipline  participated  in  the  individual  portions  of  the  survey.  Two 
members  declined  to  complete  the  Mass  Surge  Self  Assessment  of  the  survey  due  to  their 
lack  of  subject-matter  expertise  in  the  area;  leaving  the  discipline  rating  reflecting  the 
views  of  a  single  person.  A  single  person  assigning  low  ratings  is  not  definitive,  but 
those  results,  accompanied  by  the  abstentions  of  two  members,  point  to  a  lack  of 
knowledge  of  the  Mass  Surge  capability  plans  among  members  of  the  agency  responsible 
for  coordination  among  agencies. 

The  finding  —  that  the  members  of  a  discipline  with  high  levels  of  responsibility 
within  a  capability  assign  higher  scores  to  that  capability  than  other  disciplines  —  was 
also  shown  in  the  assessment  of  the  Hazardous  Materials/WMD  Response  and 
Decontamination  capability.  When  broken  down  by  discipline,  the  data  showed  the 
police  disciplines  assigned  higher  rating  than  the  average  rating  of  all  disciplines  for  each 
measure.  The  fire  discipline  assigned  a  higher-than-average  rating  in  ten  of  fourteen 
measures. 
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Figure  11.  Hazardous  Materials/WMD  Assessment  rating  by  discipline 


Figure  1 1  reinforces  a  finding  from  the  Mass  Surge  capability  assessment. 
Emergency  Management  Agency  had  difficulty  in  assessing  the  Mass  Surge  capability;  in 
the  Hazardous  Materials/WMD  Response  and  Decontamination  capability  the  data  shows 
that  the  emergency  management  personnel  routinely  gave  lower  ratings  than  other 
disciplines.  Figure  1 1  shows  the  measures  in  which  the  emergency  management 
personnel  assigned  low  ratings  that  exceeded  the  standard  deviation  for  all  assessors. 

In  regional  response  plans,  the  fire  discipline  is  assigned  responsibility  for 
implementing  plans  and  programs  to  improve  the  Hazardous  Materials/WMD  Response 
and  Decontamination  capability.  The  fire  discipline  was  found  to  provide  higher  than 
average  mean  performance  ratings.  This  should  be  expected,  as  the  group  is  most 
familiar  with  the  capability  implementation  plans.  Unfortunately,  this  assumption  may 
be  incorrect.  The  committee  that  participated  in  the  collaborative  process  also  serves  as 
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the  committee  with  direct  responsibility  for  implementing  the  Hazardous 
Materials/WMD  Response  and  Decontamination  capability.  This  committee  provided 
assessment  ratings  that  were  opposite  those  of  the  fire  discipline,  in  relation  to  the 
number  of  measures  above  or  below  average  mean  perfonnance  measure.  This  finding  is 
the  same  finding  as  was  discovered  in  the  Mass  Surge  assessment  when  it  was  examined 
by  discipline. 

The  finding  —  that  the  aggregate  ratings  of  a  group  of  individuals  from  a  single 
discipline  are  different  from  the  ratings  resulting  from  a  collaborative  effort  of  similar 
individuals  —  is  a  significant  finding.  It  may  be  a  function  of  positive  interaction  and 
learning  on  the  part  of  the  collaborative  group.  This  is  the  experience  of  the  an 
assessment  group  in  Minnesota  who  had  subgroups  complete  the  assessment  survey,  then 
come  together  as  a  group  to  discuss  the  subgroup  results  and  reach  a  consensus  rating. 
The  description  provided  by  members  of  this  group  was  that  the  consensus  effort  was  a 
learning  experience  and  the  changed  scores  reflected  knowledge  learned  about 
capabilities  in  the  consensus  process.46 

If  this  description  is  the  cause,  then  the  knowledge  of  the  experts  selected  to 
perform  the  assessment  is  called  into  question.  If  the  knowledge  about  the  capability 
assessed  is  advanced  in  the  assessment  process,  then  gained  knowledge  is  a  positive 
outcome  and  satisfies  a  goal  of  the  assessment  process.  This  finding  is  also  indicative  of 
a  lack  of  knowledge  and  the  results  of  such  an  assessment  are  of  questionable  use  to 
develop  strategic  plans  to  close  capability  gaps. 

The  assessment  disciplines  and  process  groups’  ratings  indicate  the  potential  for 
inconsistent  results  when  using  the  assessment  tool.  This  analysis  has  been  peeling  back 
the  layers  of  the  assessment  to  scrutinize  the  data.  The  data  was  first  analyzed  by  the 
assessment  process,  which  showed  that  groups  may  be  able  to  identify  a  trend  in 
capability  status.  The  data  at  the  discipline  level  began  to  show  scoring  bias  based  on 
discipline  responsibility.  The  data  at  the  discipline  level  also  showed  disagreement 
within  disciplines  about  the  status  of  capabilities. 

46  Natasha  Beluch,  “Meeting  Notes,  May  16,  2007.”  Pilot  Capabilities  Assessment  Project  (DHS: 
Washington,  DC:  Mayl6,  2007),  2. 
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G. 


CONCLUSION 


The  analysis  of  the  data  suggests  further  exploration  of  the  ability  of  the 
homeland  security  discipline  to  conduct  a  self  assessment.  If  planning  efforts  at  closing 
capability  gaps  are  to  be  informed  by  the  results  of  a  self  assessment,  then  the  potential 
for  errors  in  future  planning  efforts  exists.  Through  observations  and  interviews  with 
survey  participants,  a  trend  emerges  as  to  why  there  are  differing  opinions  on  capability 
status.  Discovery  of  this  trend  was  found  when  looking  for  data  on  the  benefits  of  the 
capability  assessment  process. 

The  following  example  suggests  two  problems.  The  first  suggestion  is  that  the 
subject-matter  experts  assembled  to  assess  the  capabilities  did  not  have  the  expertise  to 
conduct  the  assessment.  The  second  suggestion  is  that  the  assessment  process  produced 
collaboration  in  an  effort  to  solve  a  homeland  security  problem. 

In  this  example,  a  senior  emergency  management  planner  agreed  to  participate  in 
Assessment  Group  1.  Once  the  planner  received  the  assessment,  he  did  not  feel 
comfortable  providing  answers  to  the  Mass  Surge  assessment  because  of  his  lack  of 
expertise  in  the  subject.  Instead  of  returning  the  survey  incomplete,  he  passed  it  to  an 
Urban  Area  Security  Initiative  Program  Manager  for  the  Health  Emergency  Support 
Function.  This  program  manger  did  not  feel  comfortable  performing  the  assessment  and 
passed  it  on  to  the  Council  of  Governments  Medical  Surge  Hospitals  and  Health 
Subcommittee  for  completion  during  a  conference  call.  47  The  subcommittee  ultimately 
completed  the  survey  by  consensus  and  thus  became  Assessment  Group  4  for  the  Mass 
Surge  Capability  assessment. 

This  example  suggests  a  problem  with  the  knowledge  related  to  specific 
homeland  security  capability  by  a  senior  emergency  management  official.  This  lack  of 
knowledge  was  further  compounded  when  the  official  passed  the  assessment  on  to  a 
program  manager  whom  he  mistakenly  thought  had  the  requisite  knowledge  to  complete 
the  assessment.  The  program  manager  also  lacked  the  knowledge,  but  passed  on  the 
assessment  to  a  group  that  she  thought  had  the  knowledge  to  complete  the  assessment. 

47  Email  from  Participant  8  to  author,  March  28,  2007. 
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If  we  could  use  the  results  of  the  completed  assessment  to  show  that  this  group 
reached  a  valid  conclusion  about  the  status  of  the  measured  capability,  we  could 
ultimately  count  the  outcome  of  this  example  as  a  success,  but  the  data  suggests  a  broad 
diversity  of  opinions  exists  and  that  the  assessment  process  has  not  demonstrated  the 
ability  to  capture  that  diversity.  As  a  result,  that  failure  suggests  an  example  in  which  a 
lack  of  knowledge,  compounded  at  each  level  of  responsibility,  produced  results  that  may 
be  mistaken  as  valid. 

On  the  positive  side,  this  example  demonstrated  collaborative  effort  expanding 
from  a  single  assessor  to  a  group,  in  an  attempt  to  develop  an  understanding  of  the 
capability.  This  type  of  collaboration  is  necessary  to  ensure  a  diversity  of  opinion  and, 
once  knowledgeable  assessment  group  members  are  identified,  will  help  ensure  a  valid 
result. 

Did  the  different  processes  used  affect  the  outcome?  The  evidence  is  notclear. 
Figures  10  and  1 1  exhibit  the  scoring  by  groups  for  the  capabilities.  The  charts  show  the 
wide  diversity  of  opinions  on  some  measures  and  near  agreement  on  others.  These  charts 
also  suggest  a  trend  in  which  the  groups  using  the  aggregated  assessment  process  result  in 
similar  ratings  (Groups  1  and  2)  and  the  consensus  groups  provide  either  higher  or  lower 
ratings.  This  trending  is  interesting,  but  alone  it  does  not  seem  to  indicate  a  significant 
difference  of  opinion  about  the  status  of  the  capability. 
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V.  CONCLUSION 


The  Unknown 

As  we  know, 

There  are  known  knowns. 

There  are  things  we  know  we  know. 

We  also  know 

There  are  known  unknowns. 

That  is  to  say 

We  know  there  are  some  things 
We  do  not  know. 

But  there  are  also  unknown  unknowns. 

The  ones  we  don't  know 
We  don't  know. 

—February  12,  2002,  Department  of  Defense  news  briefing 

Donald  Rumsfeld48 


A.  INTRODUCTION 

This  thesis  explores  the  effect  on  outcomes  when  altering  the  process  by  which  a 
specific  assessment  methodology  is  employed.  The  research  question  became:  A  DHS 
working  group  has  proposed  a  methodology  for  performing  capability  assessments  using 
local  subject-matter  experts.  Can  the  portion  of  the  assessment  methodology  designed  to 
create  collaborative  opportunity  be  altered  without  affecting  assessment  outputs,  while 
still  creating  the  desired  additional  benefits  of  enabling  and  improving  collaboration 
among  homeland  security  professionals? 

The  small  scale  of  the  study  limits  how  the  results  of  the  assessments  can  be 
interpreted.  By  themselves,  the  surveys  mean  little.  When  combined  with  the  interviews 
of  assessment  team  participants,  however,  potentially  important  insights  into  the 
understanding  of  capabilities  at  the  local  level  are  suggested.  The  findings  and 
recommendations  in  this  chapter  discuss  this  issue  in  more  detail. 


48  Hart  Seely,  “The  Poetry  of  D.  H.  Rumsfeld,”  Slate,  April  2,  2003, 
http://www.slate.com/id/208 1042/,  accessed  July  19,  2007 


49 


B.  FINDINGS 

This  thesis  set  out  to  discover  the  level  of  effect  that  three  different  processes, 
used  to  conduct  a  self  assessment,  had  on  the  data  collected.  The  hypothesis  was  that  the 
processes  would  not  play  a  key  role  in  the  results,  and  that  data  collected  from  different 
processes  would  be  comparable  to  the  other.  The  thesis  further  hypothesized  that  the 
different  processes  would  produce  different  levels  of  collaboration  among  the 
participants,  and  that  one  process  would  be  identified  as  having  provided  more  additional 
benefits  than  the  others. 

The  results  of  the  comparative  case  study  differed  from  the  hypothesis  and 
suggest  that  the  assumption  that  “self-assessment”  is  a  reliable  and  valid  methodology  for 
performing  homeland  security  capabilities  assessment  may  be  incorrect.  Two  points  that 
support  these  findings.  First,  the  experts  —  the  “self’  —  are  not  sufficiently 
knowledgeable  of  the  capabilities  to  perform  the  self  assessment,  and  the  process  used  in 
the  assessment  does  not  matter.  In  each  of  the  three  processes  used  in  the  comparative 
case  study,  group  think  reduction  to  the  established  mean  appeared  to  be  accepted.  These 
findings  suggest  that  strategy  for  the  national  assessment  should  be  adjusted  to  fit  the 
reality.  If  we  assume  we  can  do  an  assessment,  and  the  assessment  is  faulty,  what  has 
changed  in  our  assessment  strategy  since  the  last  assessment?  The  answer  is  “nothing.” 
As  a  result,  homeland  security  may  be  doing  more  harm  than  good  by  providing  an 
illusion  of  preparedness.  The  illusion  of  preparedness  —  based  on  imprecise 
measurements  —  causes  our  community  to  make  investments  to  close  gaps  that 
ultimately  may  not  have  existed.  This  becomes  a  missed  opportunity  that  affects  our 
national  preparedness. 

The  data  suggests  that  the  homeland  security  community  in  the  National  Capitol 
Region  includes  very  few  people  who  can  claim  and  demonstrate  the  knowledge 
necessary  to  assess  the  status  of  regional  multi-discipline  efforts  to  implement  the  NPG. 
If  true,  this  finding  is  particularly  surprising  since  many  of  the  participants  in  the  surveys 
are  those  responsible  for  such  efforts.  This  lack  of  subject-matter  expertise  casts  doubt  on 
the  ability  of  a  self-assessment  tool  to  capture  an  accurate  snapshot  of  capability  status. 
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There  is  reason  to  believe  that  this  problem  is  national  in  scope.  Preliminary 
results  from  a  pilot  test  of  the  self-assessment  process  conducted  in  Minnesota  found  a 
similar  problem  in  expertise  of  the  assessment  group.  In  Minnesota,  over  one -third  of 
emergency  managers  in  the  state  are  new.  These  members  felt  they  could  use  the 
assessment  process  to  educate  the  emergency  managers.  49  This  evidence  appears  to 
support  the  finding  that  a  lack  of  expertise  exists  to  assess  capabilities. 

A  related  finding  is  that  members  of  the  homeland  security  community  were 
willing  to  use  the  results  from  this  tainted  survey  process  even  after  they  understood  the 
problems  with  the  data.  Members  felt  they  could  develop  future  plans  because  they  felt 
the  tainted  results  were  “about  right  and  when  viewed  in  comparison  with  other 
capability  results  gave  an  impression  of  status.”50,  51  This  finding  may  be  more  important 
than  the  finding  that  the  members  lacked  the  knowledge  to  conduct  the  assessment;  it 
shows  they  do  not  understand  the  importance  of  assessment.  This  finding  indicates  a 
need  to  develop  a  strategy  for  capability-based  planning  education  to  reinforce  the 
foundation  of  our  homeland  security  strategies. 

Additional  evidence,  that  the  problem  may  be  larger  than  the  region,  comes  from 
a  recent  report  issued  by  John  Hopkins  University.  UASI  homeland  security 
professionals  were  surveyed  about  compliance  with  an  understanding  of  HSPD  8. 
Recommendations  to  improve  HSPD  8  compliance,  based  on  data  collected  in  the  survey, 
called  for  enhanced  regulation  of  homeland  security  grants  and  improved  risk  assessment, 
but  notably  failed  to  include  recommendations  pertaining  to  capability  assessment.52 
This  omission  represents  the  core  of  the  assessment  problem:  The  discipline  is  immature 
and  its  members  need  guidance  and  mentoring  to  institutionalize  preparedness. 

If  a  theme  is  suggested  from  the  analysis,  it  can  be  best  defined  as  a  disconnection 
between  the  goal  of  a  capability  assessment  process  and  the  relationship  of  capabilities  to 

49Lauren  Fernandez  email  to  author,  Washington,  DC,  June  11,  2007. 

50  Interview  of  Participant  3  by  author,  Washington,  DC,  June  5,  2007. 

51  Interview  of  Participant  5  by  author,  Washington,  DC,  June  6,  2007. 

52  Amber  Gisriel,  William  Grant,  Brian  Lee,  Steve  Fruchtman,  “HSPD-8  Compliance  by  UASI 
Jurisdictions  (John  Hopkins  University  May  2007),  43 
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local  efforts.  The  inconsistent  data  on  capability  status  produced  by  the  surveys  seems  to 
indicate  a  lack  of  knowledge  by  local  homeland  security  leaders.  This  finding  is 
probably  true;  members  of  the  assessment  groups  have  been  promoted  within  their 
organizations  through  civil  service  processes  that  infer  a  level  of  expertise  within  the 
traditional  knowledge  areas  of  their  organizations.  There  is  not  a  discipline  within  this 
regional  community  of  homeland  security  capability  officials  and,  as  a  result,  no 
comparable  level  of  hierarchical  expertise  can  be  inferred. 

This  thesis  did  not  specifically  examine  what  local  officials  understood  about  the 
capabilities-based  planning  requirements  of  HSPD  8  or  whether  they  considered  the  TCL 
an  appropriate  building  block  for  preparedness.  Therefore,  a  lack  of  knowledge  by  local 
officials  cannot  be  assumed.  It  may  be  safe  to  assume  though,  that  if  local  leaders  were 
placing  the  same  priorities  upon  the  capabilities  being  measured  as  does  the  Department 
of  Homeland  Security,  it  would  follow  that  they  would  have  a  better  understanding  of 
those  capabilities.  It  is  possible  that  the  findings  of  this  comparative  case  study  are  the 
result  of  this  disconnection  in  priorities. 

The  findings  related  to  collaboration  were  not  as  hypothesized,  and  they  support  a 
possible  theory  that  members  of  the  homeland  security  community  are  focused  on 
different  issues  than  what  is  being  measured  in  the  capabilities.  Each  assessment  process 
produced  demonstrable  collaborative  efforts;  members  sought  information  about 
capability  status  from  contacts  they  believed  had  the  expertise  to  provide  capability  status 
information.  This  indicates  that  there  is  an  established  network  in  the  homeland  security 
community.  This  network  would  not  exist  without  reason. 

Assessment  Group  1  provided  an  example  of  this  collaboration  using  existing 
networks.  The  design  of  this  group  suggested  that  members  work  alone  and  complete  the 
process  with  little  or  no  collaborative  effort.  Instead,  members  of  this  group  sought  out 
other  members  to  discuss  capabilities  before  assigning  a  rating.53  The  group  knew  where 
to  look  for  expertise  and  had  the  desire,  relationships  and  authority  to  have  others 


53  Interview  of  Participant  3  by  author,  Washington,  DC,  June  5,  2007. 
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perform  the  assessment.  This  is  not  necessarily  an  example  of  lack  of  knowledge,  but  an 
example  of  the  confusion  in  the  measurement  process. 

An  example  of  the  desire  for  collaboration  of  the  assessment  groups  supported  a 
need  for  collaboration.  A  majority  of  survey  participants  stated  that  the  inclusion  of  a 
group  process  in  the  self-assessment  tool  provided  the  opportunity  to  discover  unknown 
information  about  a  capability  and  would  improve  their  effort  to  develop  solutions  and 
priorities  to  close  a  capability  gap.  A  hazardous  materials  response  official  found  that  the 
discussion  allowed  better  understanding  of  other  disciplines’  needs  to  create  capability  to 
meet  the  demands  of  a  measure.54  A  police  official  felt  that  the  opportunity  for 
collaboration  was  an  opportunity  to  engage  and  educate  support  functions  about  their 
roles  in  creating  a  capability.55 

To  summarize  the  results  of  this  comparative  case  study,  the  most  important 
suggestion  is  that  local  officials  cannot  use  this  assessment  tool  to  provide  the  snapshot  of 
capability  status  desired  by  the  Department  of  Homeland  Security.  This  is  not  necessarily 
because  of  faults  in  the  assessment  tool,  but  because  of  the  knowledge  base  of  the  local 
officials.  An  examination  of  the  results  suggests  an  undercurrent  of  homeland  security 
activities  that  the  assessment  tool  does  not  capture:  Local  officials  are  focused  on 
homeland  security  issues  that  are  not  the  same  as  the  assessment  tool  measures,  and  they 
lack  an  appreciation  for  the  significance  of  assessment. 

C.  RECOMMENDATIONS 

The  comparative  case  study  conducted  in  this  thesis  suggests  insight  into  the 
limits  of  the  capability-assessment  process  in  the  homeland  security  discipline.  The 
findings  suggest  an  understanding  of  the  status  of  a  capability  was  unlikely  to  be 
gained  directly  from  this  assessment  methodology. 

These  results  suggest  a  need  to  change  strategy  in  our  homeland  security 
capabilities-assessment  processes.  Current  assessment  strategies  are  designed  to 

54  interview  of  Participant  13  by  author,  Washington,  DC,  June  8,  2007. 

55  Interview  of  Participant  14  by  author,  Washington,  DC,  June  5,  2007. 
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document  organizational  accomplishments,  but  do  little  to  reveal  the  effect  of  those 
accomplishments.  An  example  of  an  area  that  has  an  existing  strategy  that  may  be 
effective  is  in  the  fire  service.  The  consensus  standards  issued  by  the  National  Fire 
Protection  Association  create  a  national  model  of  minimum  fire  protection  standards 
calibrated  to  community  size  and  resources.  Through  minimum  training  standards,  local 
fire  chiefs  and  firefighters  develop  knowledge  of  these  standards.  Because  the 
knowledge  of  the  firefighting  capability  is  standardized  through  an  accepted  consensus 
process,  the  standards  have  validity  that  the  current  assessment  process  lacks. 

These  standards  create  levels  of  understanding  that  are  national  in  scope.  For 
example,  fire  fighters  with  basic  fire  training  anywhere  in  the  county  learn  the  same  fire 
prevention  strategies.56  Fire  officers  around  the  country  receive  the  same  information 
about  response  times  and  safety  standards.57  This  widespread  knowledge  is  missing  from 
the  homeland  security  discipline.  The  language  of  the  discipline  has  not  yet  been  fully 
developed. 

A  new  assessment  strategy  would  place  a  higher  priority  on  the  understanding  of 
homeland  security  capability  than  on  the  assessment  of  homeland  security  capabilities. 
The  new  strategy  should  focus  on  identifying  what  is  working  within  homeland  security. 
The  needed  changes  require  the  development  of  concepts  for  asking  the  right  questions, 
questions  that  will  create  knowledge  of  local  priority-setting  processes  and  planning 
efforts. 

The  results  of  these  assessments  could  then  be  used  to  develop  a  national  picture 
of  local  capabilities  using  a  different  matrix  than  the  NPG.  A  new  matrix  would  provide 
an  overview  of  ah  preparedness  efforts,  not  just  the  ones  associated  with  HSPD  8.  This  is 
necessary  because  the  federal  direction  about  preparedness  activities  is  conflicted. 

Not  ah  federal  planning  efforts  align  with  the  NPG.  Specifically,  HSPD  5  and  the 
National  Response  Plan  designate  additional  capabilities,  planning  assumptions  or 

56  NFPA  1001,  Standard  for  Fire  Fighter  Professional  Qualifications,  2008  Edition,  Annex  A 
Explanatory  Material,  6.5.2  (NFPA<  Quincy,  MA). 

57  NFPA  1021,  Standard  for  Fire  Officer  Professional  Qualifications,  2003  Edition,  Chapter  7.6 
Emergency  Services  Delivery  and  7.7  Flealth  and  Safety  (NFPA:  Quincy,  MA). 
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separate  functions  that  create  confusion  when  local  authorities  try  to  detennine 
implementation  priorities  for  capability-based  planning  efforts.  This  misalignment  may 
help  to  explain  why  local  officials  have  developed  systems  that  are  not  captured  in  the 
capability  assessment. 

For  example,  HSPD  5  addresses  two  of  the  elements  listed  on  the  TCL:  on-site 
and  emergency  operations  center  management.  The  HSPD  5  scenarios  match  only  5  of 
the  15  listed  under  the  TCL  or  HSPD  8:  hurricane,  earthquake,  hazardous  materials  and 
nuclear.58  The  National  Response  Plan  designates  responsibilities  among  federal 
agencies  for  federal  response  to  significant  events.  The  NRP  designates  15  Emergency 
Support  Functions  to  accomplish  response  tasks.  “Four  of  the  15  ESF  groupings  do  not 
align  with  the  elements  of  the  TCL:  transportation,  public  works  and  engineering,  energy 
and  external  affairs.”59 

To  develop  the  necessary  snapshot  of  capability  status,  a  new  assessment  strategy 
will  have  to  assess  more  than  HSPD  8  and  the  TCL.  The  measurement  must  include  all 
preparedness  efforts  that  local  authorities  are  trying  to  implement.  These  efforts  will 
provide  the  needed  data  to  detennine  capability  status. 

1.  Leadership 

This  thesis  has  identified  problems  with  the  levels  of  homeland  security 
knowledge  and  recommended  improved  education  in  capabilities-based  planning  for 
homeland  security  leaders.  The  thesis  has  also  argued  that  the  NPG,  when  used  as  a 
measuring  stick  for  these  professionals,  may  be  the  wrong  measure. 

This  is  similar  to  the  situation  Bratton  faced  when  he  took  over  the  New  York 
City  Police  Department.  His  commanders  did  not  believe  there  were  crime  problems  in 
the  subway  system  because  the  statistics  told  them  only  3  percent  of  the  city’s  major 
crimes  took  place  in  the  subway.  Bratton  felt  the  commanders’  assessment  of  crime  was 

58  Chris  Bellavita,  Complex  Adaptive  Lecture,  July  17,  2007,  quoting  White  House,  Homeland 
Security  Presidential  Directive/HSPD  5  (Washington,  DC:  The  White  House,  February  28,  2003),  22. 

59  Bellavita  Lecture,  July  17,  2007,  quoting  National  Response  Plan  (Washington,  DC:  Department  of 
Homeland  Security,  March  2004):  19. 
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incorrect.  Bratton  heard  the  complaints  of  citizens  about  conditions  in  the  subways  and 
dispatched  his  commanders  to  ride  the  subway.  In  a  matter  of  weeks,  police  strategy 
changed  to  focus  on  reducing  crime  on  the  subways,  because  the  police  commanders 
recognized  the  problems  with  their  previous  assessments  and  adjusted  their  policing 
strategies  accordingly.60 

By  viewing  the  tested  capabilities-assessment  process  as  crime  statistics  in  the 
subway  system  in  this  example,  we  can  see  that  we  are  maintaining  the  status  quo  in 
regards  to  our  knowledge.  If  we  require  our  leadership  to  participate  in  an  educational 
assessment  process  that  focuses  on  understanding  the  existing  capabilities  and  needs,  we 
should  be  able  to  develop  leadership  momentum  for  changes  to  the  methodologies  we  use 
to  examine  capabilities.  Riding  the  subway  was  the  educational  assessment  process  for 
the  police  commanders. 

The  assessment  process  becomes  the  tipping  point.  Leaders  who  cannot  show 
knowledge  of  their  capabilities  —  and  defend  those  capabilities  as  being  appropriate  to 
their  local  needs  —  should  be  replaced  with  leaders  who  can.  The  key  to  this  tipping 
point  is  that  it  is  tied  to  the  leaders’  understanding  of  their  capabilities  as  related  to  local 
needs,  not  to  the  NPG. 

2.  New  Assessment  Tactics 

The  development  of  tactics  for  gaining  an  understanding  of  what  local  homeland 
security  efforts  are,  and  how  they  are  developing,  can  be  very  simple  and  straightforward. 
A  large,  relatively  untapped  body  of  capability  status  evidence  exists.  This  evidence  is 
related  to  actual  emergencies  and  all  levels  of  response  to  the  incident.  There  is  also  a 
large  body  of  evidence  related  to  exercises. 

A  new  assessment  tactic  could  the  comparison  of  actions  related  to  a  past  incident 
to  the  same  incident  scenario  today.  The  actual  incident  could  be  studied  through  after- 


60  W.  Chan  Kim  and  Renee  Mauborgne,  Blue  Ocean  Strategy  (Boston,  MA:  Harvard  University  Press, 
2005),  153. 
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action  reports  and  interviews  with  key  decision  makers.  The  current  scenario  can  be 
assessed  through  interviews  at  multiple  levels  within  an  organization.  The  result  is  to 
understand  the  effect  of  an  investment. 

For  instance,  the  Arlington  County  After  Action  Report  on  the  9/11  Pentagon 
Attack  can  be  used  as  a  basis  for  examining  an  actual  response.  Asking  the  local  police 
and  fire  officials,  “What  would  be  different  if  the  attack  were  to  occur  today?”  the 
discussion  would  continue  until  they  gained  an  understanding  of  the  effect  of  any  added 
capabilities. 

For  example,  would  the  dollars  invested  in  urban  search  and  rescue  teams  have 
saved  any  lives  in  the  same  scenario?  If  the  answer  is  no,  then  the  net  effect  of  the 
investment  is  zero.  If  the  answer  is  yes,  then  there  is  a  gain  from  the  investment.  In 
either  case,  questions  about  why  the  urban  search  and  rescue  investment  received  funding 
and  how  the  investment  was  implemented  will  provide  useful  data.  This  data  will  allow  a 
broader  examination  of  influences  on  the  homeland  security  decisions  of  local 
governments. 

To  examine  a  capability  that  does  not  have  an  incident  parallel,  we  can  use 
exercise  data.  A  comparative  case  study  of  exercises  and  response  polices  to  radioactive 
dirty  bombs  may  reveal  that  the  baseline  policy  for  a  region  on  9/11  would  have 
consisted  of  rapid  response  by  all  disciplines  to  perform  rescue,  removal  and 
decontamination  of  potentially  exposed  persons.  A  comparative  case  study  of 
investments  related  to  radiation  response  may  show  multiple  programs  to  improve 
personal  protection,  detection  and  decontamination. 

A  comparative  case  study  of  recent  exercises  and  current  response  polices  may 
indicate  that  the  effect  of  response  polices  remains  the  same  as  on  9/1 1 :  rapid  response  by 
all  disciplines  to  perform  rescue,  removal  and  decontamination  of  potentially  exposed 
persons.  At  first  blush,  it  would  seem  that  no  gain  has  been  accomplished  through  the 
investments  because  the  effect  of  the  response  is  the  same  before  and  after  the 
investment. 
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Questioning  of  members  in  various  disciplines  should  take  place  to  identify  other 
areas  of  benefit  from  the  investments.  Arguments  can  be  made  that  the  investments  will 
result  in  rapid  identification  of  a  radiation  dirty  bomb  incident  and  that  this  early 
notification  will  allow  notification  to  hospitals,  utilities  and  transportation  disciplines  to 
protect  critical  infrastructure.  Early  notification  will  cause  responders  to  take  specific 
actions  that  will  improve  their  safety,  and  such  increased  detection  capability  creates 
opportunity  for  incident  prevention. 

These  examinations  create  opportunities  for  tipping  point  leadership.  The 
examination  creates  opportunity  for  self  examination  among  leaders.  For  example,  if  a 
fire  chief  has  invested  heavily  in  Urban  Search  and  Rescue  and,  through  this  process,  sees 
that  the  investment  did  not  affect  the  outcome  of  an  incident,  then  the  fire  chief  has  an 
opportunity  to  change  the  way  he  will  evaluate  future  investments.  Having  to  defend  the 
effect  of  investments  in  radiation-detection  capabilities  can  cause  the  fire  chief  to  develop 
a  keener  understanding  of  the  complexities  of  a  radiation  response.  This  knowledge  can 
change  behavior  and  become  a  tipping  point  for  a  better  understanding  of  homeland 
security  investments.  Through  improved  investing,  the  leader  can  make  changes  that 
increase  preparedness.  Leaders  who  continue  to  make  investments  that  would  not 
improve  an  outcome  will  soon  find  themselves  and  their  organizations  left  behind  in  the 
funding  processes. 

3.  Areas  for  Future  Research 

This  thesis  identified  an  area  of  strength  within  the  capability  assessment  process. 
This  strength  is  the  desire  and  willingness  of  the  homeland  security  community  to  use 
collaborative  efforts  to  solve  problems.  The  recommendation  made  in  this  thesis  is  to 
utilize  areas  of  strength  in  homeland  security  and  use  those  strengths  as  the  basis  for 
future  efforts  to  learn  about  homeland  security  capabilities. 

Continuing  to  create  the  academic  field  of  homeland  security  is  one  of  the  keys  to 
implementing  an  understanding  of  homeland  security  and  establishing  national  standards 
and  policies  for  homeland  security.  The  tactic  described  to  compare  and  contrast  past 
and  current  capability  requires  expertise  levels  sufficient  to  develop  an  understanding  of 
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the  effects  of  changes  in  capabilities.  This  thesis  has  shown  there  are  problems  with 
current  levels  of  expertise;  overcoming  this  issue  will  be  challenging. 

Additionally,  an  improved  understanding  of  effects-based  planning  related  to 
homeland  security  is  needed.  The  military  is  using  effects-based  planning  for 
warfighting  capabilities.61  These  efforts  should  be  examined  for  transferable  knowledge 
to  the  homeland  security  discipline. 

D.  CONCLUSION 

This  thesis  evolved  because  of  a  perception  that  the  capability  assessment 
requirements  being  heaped  on  local  government  were  not  adding  value  to  homeland 
security  efforts.  The  evolution  of  possible  solutions  to  this  problem  sought  to  add  value 
and  make  improvements  to  existing  strategies  in  capabilities  assessments.  Ultimately  a 
promising  tool  was  developed  and  tested.  The  analysis  of  the  testing  suggests  that  the 
tool  may  not  function  as  designed.  This  is  not  necessarily  because  the  tool  was 
ineffective,  but  because  existing  homeland  security  strategies  have  not  focused  on 
developing  the  understanding  of  local  capabilities  and  needs  necessary  to  assess  those 
capabilities. 

The  Department  of  Homeland  Security  stands  to  gain  from  investment  in 
understanding  of  total  capability  by  local  government  through  participation  in  this 
process.  The  migration  of  perspective  —  from  national  capability  to  a  regional  capability 
to  local  capability  —  is  necessary  to  make  substantial  and  lasting  gains  in  homeland 
security  preparedness.  This  perspective  is  what  is  missing  from  current  efforts  and  we 
should  focus  on  causing  change  for  future  efforts. 


61  Paul  Davis  and  Gail  Zellman,  Effects  Based  Operations  (Santa  Monica,  CA:  RAND,  2001),  1. 
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APPENDIX  A:  MEASURES  -  MASS  SURGE  CAPABILITY 


Measure 

Description 

1 

Our  region's  plans  specify  the  criteria  and  process  for  activating  Medical  Surge  procedures. 

2 

Our  region’s  plans  for  Medical  Surge  are  based  on  a  formal  assessment  of  risks  and 
vulnerabilities. 

3 

Our  region's  plans  address  the  use  of  existing  facilities  in  support  of  Medical  Surge  (e.g., 
hospitals,  clinics,  extended  care  facilities). 

4 

Our  region’s  plans  address  the  identification  and  setting  up  of  additional  facilities  (e.g., 
provision  of  personnel,  equipment,  pharmaceuticals). 

5 

Our  region's  plans  address  cultural  characteristics  of  populations  to  be  treated  (e.g.,  religious 
needs,  language  barriers). 

6 

Our  region's  plans  address  the  treatment  requirements  of  special  needs  populations. 

7 

Our  region's  plans  address  patient  and  resource  transportation  (e.g.,  identification  and 
availability  of  traditional  and  non-traditional  resources). 

8 

Our  region's  plans  address  facility  based  evacuation  (e.g.,  identification  of  receiving  facilities, 
coordination  of  transportation  assets). 

9 

Our  region's  plans  address  the  treatment  of  Medical  Surge  response  personnel,  site  staff,  and 
their  families  (e.g.,  medical  needs,  stress  management  strategies). 

10 

Our  region’s  plans  address  public  information  and  communications  requirements  relevant  to 
Medical  Surge  (e.g.,  dissemination  of  accurate,  timely,  accessible  information  to  public, 
media,  support  agencies). 

11 

Nonprofit  organizations/NGO’s  are  actively  engaged  in  your  region’s  development  of  plans  for 
Medical  Surge  (e.g.,  the  American  Red  Cross,  Salvation  Army,  faith-based  organizations). 

12 

Our  region's  emergency  transport  and  tracking  systems  are  interoperable  with  national  and 
Department  of  Defense  systems. 

13 

Our  region’s  supply  inventories  and  reserves  are  adequate  to  support  of  Medical  Surge. 

14 

Our  region  maintains  databases  that  track  the  status  of  resources  (e.g.,  medications,  medical 
professionals)  available  to  support  Medical  Surge. 

15 

Our  region  currently  has  (or  has  access  to)  the  personnel  it  needs  to  support  Medical  Surge 
(e.g.,  medical  service  providers,  patient  transporters,  security  personnel). 
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16 

Our  region  has  a  robust  program  for  recruiting  volunteers  to  support  Medical  Surge. 

17 

Our  region  has  developed  a  method  for  managing  spontaneous  volunteers  and  donated 
resources  in  support  of  Medical  Surge  (That  is,  is  there  a  system  to  intake  volunteers,  assign 
responsibilities  and  direct  resources  where  they  are  most  needed). 

18 

Our  region  currently  has  (or  has  access  to)  the  equipment  it  needs  to  support  Medical  Surge 
(e.g.,  medical  equipment,  transportation  vehicles). 

19 

Our  region  is  able  to  ensure  safety  and  security  at  the  facilities  it  intends  to  use  to  support 
Medical  Surge  (That  is,  can  your  region  determine  and  ensure  the  structural  integrity, 
capability,  and  physical  security  of  its  Medical  Surge  facilities). 

20 

Our  region  is  able  to  ensure  the  self-sufficiency  of  the  facilities  it  intends  to  use  to  support 
Medical  Surge  (e.g.,  with  respect  to  communications,  power). 

21 

Our  region  has  the  information  resources  required  to  project  the  demand  for  Medical  Surge 
(e.g.,  how  many  people  will  need  treatment,  how  long  it  will  take  to  secure  facilities). 

22 

Our  region  has  addressed  the  credentialing  requirements  of  Medical  Surge. 

23 

Our  region  have  scalable  patient  tracking  systems. 

24 

Our  region  has  exercised  its  ability  to  implement  Medical  Surge  (e.g.,  the  provision  of 
medication,  follow-up  care,  securing  of  facilities). 
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APPENDIX  B:  MEASURES  -  HAZARDOUS  MATERIALS  /  WMD 
RESPONSE  AND  DECONTAMINATION  CAPABILITY 


Measure 

Description 

1 

Our  region’s  plans  specify  the  criteria  for  activating  WM D/Hazardous  Materials  Response  and 
Decontamination  procedures. 

2 

Our  region’s  plans  for  WMD/Hazardous  Materials  Response  and  Decontamination  are  based  on  a  formal 
assessment  of  risks  and  vulnerabilities. 

3 

Our  region’s  plans  for  WMD/Hazardous  Materials  Response  and  Decontamination  address  responder 
safety  (e.g.,  accurate  identification  of  hazards,  appropriately  certified  responders,  proper  PPE). 

4 

Our  region’s  public  safety  and  HazMat  personnel  are  trained  at  various  levels  (e.g.,  hazardous  awareness, 
hazardous  materials  operations,  hazmat  technician,  hazmat  specialist,  hazardous  materials  management). 

5 

Our  region’s  HazMat  personnel  are  equipped  and  trained  for  weather  prediction  and  hazard  plume 
modeling. 

6 

Our  region’s  plans  for  WMD/Hazardous  Materials  Response  and  Decontamination  address  substance 
identification  equipment  (e.g.,  bases,  vapors,  liquids,  solids,  biologicals  like  white  powder). 

7 

Our  region’s  plans  for  WMD/Hazardous  Materials  Response  and  Decontamination  address  personnel 
needs  (e.g.,  work/rest  cycles,  medical,  psychological,  financial  assistance,  etc). 

8 

Our  region’s  plans  address  information  sharing  requirements  relevant  to  WMD/Hazardous  Materials 
Response  and  Decontamination  (e.g.,  dissemination  of  accurate,  timely,  accessible  information  to  public, 
media,  support  agencies,  etc.). 

9 

Our  region  currently  has  (or  has  access  to)  the  personnel  it  needs  to  support  WMD/Hazardous  Materials 
Response  and  Decontamination  (e.g.,  safety  officers,  HazMat  personnel,  medical  specialists). 

10 

Our  region  currently  has  (or  has  access  to)  the  equipment  it  needs  to  support  WMD/Hazardous  Materials 
Response  and  Decontamination  (e.g.,  PPE,  medical  equipment,  WMD  detectors). 

11 

Our  region  has  information  tracking  systems  relevant  to  WMD/Hazardous  Materials  Response  and 
Decontamination  (e.g.,  hot  zones  locations,  real-time  hazard  databases,  decontaminated  persons 
updates). 

12 

Our  region  is  able  to  ensure  the  safety  and  security  at  the  scene  and  support  facilities  it  intends  to  use  for 
WMD/Hazardous  Materials  Response  and  Decontamination  (That  is,  can  your  region  determine  and 
ensure  the  structural  integrity,  capability,  and  physical  security  of  its  HazMat  Management  facilities.). 

13 

Our  region  is  able  to  ensure  the  self-sufficiency  of  its  WMD/Hazardous  Materials  Response  and 
Decontamination  operations  (e.g.,  with  respect  to  communications,  power). 

14 

Our  region’s  plans  address  the  demobilization  of  WMD/Hazardous  Materials  Response  and 

Decontamination  (e.g.,  debrief  personnel,  repackage  equipment). 

15 

To  what  extent  has  your  region  exercised  its  ability  to  implement  WMD/Hazardous  Materials  Response 
and  Decontamination  procedures  (e.g.,  exercised  response  to  pre-identified  hazards,  decontaminated 
victims). 
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INITIAL  DISTRIBUTION  LIST 


1 .  Defense  Technical  Infonnation  Center 
Ft.  Belvoir,  Virginia 

2.  Dudley  Knox  Library 
Naval  Postgraduate  School 
Monterey,  California 

3.  Fire  Chief  Dennis  Rubin 

District  of  Columbia  Fire  and  Emergency  Medical  Services 
Washington,  District  of  Columbia 

4.  Fire  Chief  Nicholas  Finamore 
Allentown  Road  Volunteer  Fire  Department 
Fort  Washington,  Maryland 

5.  Chief  Cathy  Lanier 
Metropolitan  Police  Department 
Washington,  District  of  Columbia 


71 


